Systems and methods for vibration imaging and sensing - Patents.com

Abstract

[0001] The present disclosure generally relates to systems and methods for imaging and sensing vibrations of an object. In one implementation, a system for detecting vibrations is provided, the system including a sensor having a plurality of pixels, each pixel including at least one photosensitive element. The sensor is configured to output a signal related to illumination on each pixel. The system also includes at least one processor configured to receive the output signal from the sensor and detect one or more polarities based on the output signal. The at least one processor is also configured to determine a timestamp associated with the output signal, analyze the timestamp and the polarity to detect vibrations of an object within a field of view of the sensor, and determine one or more frequencies of the detected vibrations.

Description

[Technical field]

[0001] The present disclosure generally relates to the field of imaging and sensing properties of objects. More particularly, but not exclusively, the present disclosure relates to computer-implemented imaging systems and methods for detecting and analyzing vibrations of objects. The present disclosure additionally relates to the implementation and use of pixel-based sensors for vibration detection and analysis. The sensors and techniques disclosed herein may be used in a variety of applications and systems, such as the analysis of moving machine elements such as rotating turbines or shafts, systems for monitoring engine or vehicle health, and other applications and systems that benefit from accurate and efficient vibration analysis. [Background technology]

[0002] Existing vibration sensors, such as accelerometers, directly contact the device or element whose vibration is being measured. Such sensors change the mechanical properties of these devices and therefore lack accuracy. They also have other drawbacks, such as the inability to make measurements when the vibrating arm or element is hot, inaccessible, or too small, making contact sensors inapplicable.

[0003] In some applications, optical fibers with non-coherent light (such as light emitting diodes (LEDs)) and photodetectors may be used to provide a non-contact vibration detector. However, such implementations do not have the ability to resolve vibrations across space. Furthermore, accuracy is generally limited to the sensitivity of the photodetector to the varying illumination of the non-coherent light.

[0004] Without sufficient precision, it is difficult to detect slow damping of machine elements and machines, since slow changes in vibration patterns (e.g., frequency, amplitude, etc.) may go undetected. Furthermore, without spatial resolution, machines made up of multiple parts cannot be accurately monitored when different parts of the machine exhibit different vibration patterns. Summary of the Invention

[0005] Therefore, vibration sensing solutions with high accuracy (eg, based on high temporal resolution) and improved spatial resolution are desired.

[0006] The embodiments of the present disclosure provide computer-implemented systems, methods, and non-transitory computer storage media that address the above-mentioned shortcomings. The embodiments of the present disclosure include embodiments for vibration detection and analysis with high time resolution. In addition, the embodiments of the present disclosure provide systems and methods for vibration detection and analysis with the ability to distinguish frequencies and amplitudes across space. These and other features, aspects, and embodiments of the present disclosure are described in detail herein with reference to Figures 3-9.

[0007] In some embodiments, the generated data may include multiple frequencies and amplitudes, allowing a two-dimensional image of the vibrations in a scene to be generated, for example as a vibration map. An example of such a map is shown in Figure 9 and described further below. Additionally, embodiments of the present disclosure may be used to monitor machine elements and objects, such as factory machines, vehicles, etc., so that damping of components may be detected early by identifying changes in vibration patterns with greater accuracy than existing techniques.

[0008] Embodiments of the present disclosure may use natural or artificial illumination of the object undergoing vibration and / or project an electromagnetic pulse (whether uniform or structured) to generate a reflection of the object at the sensor. In some embodiments, the sensor may include multiple pixels and index the received reflection by the location of the pixel receiving the reflection. Thus, embodiments of the present disclosure may calculate and map the vibration and amplitude to a two-dimensional projection of the scene based on the spatial coordinates of the pixel.

[0009] To process the frequency with higher accuracy, the embodiments of the present disclosure may apply an event counter to discard spurious events, so that vibrations can be detected while rejecting noise to improve the accuracy of the frequency calculation. Thus, the embodiments of the present disclosure may address some of the shortcomings of existing techniques, as described above.

[0010] In some embodiments, the temporal resolution can be further improved by using event-based sensors. Such sensors can capture events in a scene based on changes in illumination at a particular pixel exceeding a threshold. The sensors can operate asynchronously and detect events from scene reflections while reducing the amount of data generated. Thus, the temporal resolution can be improved.

[0011] As explained above, data reduction through the use of event-based sensors may allow the rate of light sampling at each pixel to be increased from, for example, 30 times per second or 60 times per second (i.e., the frame rate of a typical CMOS image sensor) to higher rates such as 1,000 times per second, 10,000 times per second, and more. The higher rate of light sampling improves the accuracy of the frequency calculation compared to existing techniques.

[0012] Embodiments of the present disclosure may also provide spatial resolution of frequency, unlike existing approaches. For example, events may be detected at individual pixels (and thus frequency may also be calculated for individual pixels) rather than relying on general changes in illumination or other properties across the image sensor. Moreover, the use of event-based sensors further improves this spatial resolution.

[0013] The embodiments of the present disclosure may detect amplitude and frequency in an image. For example, the embodiments of the present disclosure may normalize the calculated frequency by the number of detected events to determine the amplitude. Thus, the accuracy of the calculated amplitude may improve as the accuracy of the frequency increases. Furthermore, since events are detected at individual pixels, the amplitude may generally be determined across space, not across the entire sensor. In addition, if an event-based sensor is used, the amplitude may be detected with greater accuracy.

[0014] According to one aspect of the disclosure, a system for detecting vibrations is provided that includes a sensor. The sensor may include a plurality of pixels, each pixel including at least one photosensitive element. The sensor may be configured to output a signal associated with illumination on each pixel. The system may further include at least one processor, configured to receive an output signal from the sensor, determine one or more changes in illumination based on the output signal, the one or more changes including polarity information, the polarity information indicating whether the change in illumination is associated with an increase in illumination or a decrease in illumination, determine a timestamp associated with the one or more changes, analyze the timestamp and the polarity information to detect vibrations of an object within a field of view of the sensor, and determine one or more frequencies of the detected vibrations.

[0015] In such an embodiment, the at least one processor may be configured to determine a positive polarity when the change in illumination indicates an increase in illumination and to determine a negative polarity when the change in illumination indicates a decrease in illumination.

[0016] In any of these embodiments, the change in illumination may be associated with a set of pixels of the sensor, for example, one or more frequencies may be associated with the set of pixels based on the change in illumination.

[0017] In any of these embodiments, the at least one processor may be configured to determine the time stamp by extracting multiple time stamps from a readout system associated with the sensor. For example, the readout system may encode one or more time stamps based on an Address-Event Representation (AER) protocol. Additionally or alternatively, the readout system may encode an address that identifies an associated pixel for each output signal.

[0018] In any of these embodiments, the at least one processor may be configured to analyze the time stamps by calculating a time interval between one or more polarity changes using the multiple time stamps, and determine one or more frequencies based on the calculated time intervals. In such an embodiment, the at least one processor may be further configured to calculate an average or median value and a deviation associated with the calculated time intervals, and determine one or more frequencies based on an inverse of the average value when the average value is calculated, or determine one or more frequencies based on the median value when the median value is calculated, and validate the one or more frequencies when the deviation falls below a validation threshold. For example, the validation threshold may be about 1% to 10% of the average or median value. The validation threshold may be greater than or equal to 1% and less than or equal to 10% of the average or median value, as the case may be.

[0019] In any of these embodiments, the at least one processor may be configured to analyze output signals associated with one or more of the plurality of pixels to determine one or more amplitudes of the vibrations. For example, the at least one processor may be configured to apply a spatial filter to the output signals from the plurality of pixels and determine the one or more amplitudes based on the applied spatial filter. Additionally or alternatively, the at least one processor may be configured to perform two-dimensional shape tracking on the output signals to determine a shape of the object undergoing the vibrations and to determine the one or more amplitudes. For example, the at least one processor may be configured to perform two-dimensional shape tracking by applying an iterative closest point algorithm.

[0020] Additionally or alternatively, the at least one processor may be configured to determine a number of illumination changes from at least one of the plurality of pixels within a unit time and normalize the number of changes by the determined frequency of the one or more frequencies associated with the at least one pixel to determine the one or more amplitudes. For example, the unit time may be about 1 ms or less. The time unit may be 1 ms or less.

[0021] In any of these embodiments, the system may further include an external light source that projects light onto an object within the field of view of the sensor. For example, the external light source may project uniform light or structured light. The external light source may project light having a constant illumination. The external light source may project light including a structured illumination pattern. In such an embodiment, the external light source may project light at an oblique angle relative to at least one surface of the object.

[0022] In any of these embodiments, the at least one processor may be configured to filter the one or more illumination changes to remove spurious changes.

[0023] In any of these embodiments, the at least one processor may be configured to reject timestamps that differ by more than or equal to a time threshold, for example, the time threshold used to reject timestamps may be 1 ms or greater.

[0024] In some embodiments, a system for detecting vibration may include a sensor. The sensor may include a plurality of photosensitive elements configured to detect illuminance on the elements, a plurality of change detectors configured to output a signal when a change in the detected illuminance on one or more of the photosensitive elements exceeds a threshold, and at least one processor. The at least one processor may be configured to receive the output signal, determine a polarity based on the output signal, the polarity indicating whether the illumination is increasing or decreasing, determine a timestamp associated with the output signal from the plurality of change detectors, analyze the timestamp and the polarity to detect vibration of an object within a field of view of the sensor, and determine one or more frequencies of the detected vibration.

[0025] In such an embodiment, positive polarity is associated with increased illumination and negative polarity is associated with decreased illumination.

[0026] In any of these embodiments, a polarity is associated with a set of pixels, each of which may include at least one of a plurality of photosensitive elements, and thus, one or more frequencies may be associated with a set of pixels based on the polarity.

[0027] In any of these embodiments, the at least one processor may be configured to determine the time stamp by extracting multiple time stamps from a readout system associated with the sensor. For example, the readout system may encode one or more time stamps based on an Address-Event Representation (AER) protocol. Additionally or alternatively, the readout system may encode an address that identifies an associated pixel for each output signal.

[0028] In any of these embodiments, the at least one processor may be configured to analyze the time stamps to calculate time intervals between one or more polarity changes using the multiple time stamps, and determine one or more frequencies based on the calculated time intervals. In such embodiments, the at least one processor may be further configured to calculate an average or median value and a deviation associated with the calculated time intervals, determine one or more frequencies based on an inverse of the average value when the average value is calculated, or determine one or more frequencies based on an inverse of the median value when the median value is calculated, and validate the one or more frequencies when the deviation falls below a validation threshold. For example, the validation threshold may be about 1% to 10% of the average or median value. The validation threshold may be greater than or equal to 1% and less than or equal to 10% of the average or median value, as the case may be.

[0029] In any of these embodiments, the at least one processor may be configured to analyze the output signal to determine one or more amplitudes of the vibrations. For example, the at least one processor may be configured to apply a spatial filter to the output signal and determine the one or more amplitudes based on the applied spatial filter. Additionally or alternatively, the at least one processor may be configured to perform two-dimensional shape tracking on the output signal to determine a shape of the object subjected to the vibrations and to determine the one or more amplitudes. For example, the at least one processor may be configured to perform two-dimensional shape tracking by applying an iterative closest point algorithm.

[0030] Additionally or alternatively, the at least one processor is configured to determine a number of illumination changes from at least one of the plurality of pixels within a unit time and normalize the number of changes by the determined frequency of the one or more frequencies associated with the at least one pixel to determine the one or more amplitudes. For example, the unit time may be about 1 ms or less. The time unit may be 1 ms or less.

[0031] In any of these embodiments, the system may further include an external light source configured to project light onto an object within the field of view of the sensor. For example, the external light source may project uniform light or structured light. Uniform light may refer to light having a constant illumination. Structured light may refer to light that includes a structured illumination pattern. In such an embodiment, the external light source may project light at an oblique angle relative to at least one surface of the object.

[0032] In any of these embodiments, the at least one processor may be configured to filter one or more output signals to remove spurious signals.

[0033] In any of these embodiments, the at least one processor may be configured to reject timestamps associated with time differences equal to or greater than a time threshold. For example, the time threshold may be about 1 ms or greater. The time threshold may be 1 ms or greater.

[0034] According to one aspect of the disclosure, a method for detecting vibration is provided, which may include receiving, from a sensor including a plurality of pixels, an output signal associated with illumination of one or more of the pixels, each of the pixels including at least one photosensitive element, determining a polarity indicative of whether illumination on the one or more pixels is increasing or decreasing based on the output signal, determining a timestamp associated with the output signal, analyzing the timestamp and the polarity to detect vibration of an object within a field of view of the sensor, and determining one or more frequencies of the detected vibration.

[0035] In such an embodiment, the method may further include determining a positive polarity when illumination on one or more pixels is increasing, and determining a negative polarity when illumination on one or more pixels is decreasing.

[0036] In any of these embodiments, the output signal may be associated with a set of pixels of the plurality of pixels. Thus, one or more frequencies may be associated with the set of pixels based on the output signal.

[0037] In any of these embodiments, the method may further include determining the time stamp by extracting a plurality of time stamps from a readout system associated with the sensor. In such an embodiment, the readout system may encode one or more time stamps based on an Address-Event Representation (AER) protocol. Additionally or alternatively, the readout system may encode an address that identifies an associated pixel for each output signal.

[0038] In any of these embodiments, the method may further include analyzing the timestamps by calculating a time interval between one or more polarity changes using the multiple timestamps, and determining one or more frequencies based on the calculated time intervals. In such an embodiment, the method may further include calculating an average or median value and a deviation associated with the calculated time intervals, determining one or more frequencies based on an inverse of the average value when an average value is calculated, or determining one or more frequencies based on an inverse of the median value when a median value is calculated, and validating the one or more frequencies when the deviation falls below a validation threshold. For example, the validation threshold may be about 1% to 10% of the average or median value. The validation threshold may be greater than or equal to 1% and less than or equal to 10% of the average or median value, as the case may be.

[0039] In any of these embodiments, the method may further include analyzing the output signal to determine one or more amplitudes of the vibrations. In such embodiments, the method may further include applying a spatial filter to the output signal and determining the one or more amplitudes based on the applied spatial filter. Additionally or alternatively, the method may further include performing two-dimensional shape tracking on the output signal to determine a shape of the object subjected to the vibrations and to determine the one or more amplitudes. For example, the method may further include performing two-dimensional shape tracking by applying an iterative closest point algorithm.

[0040] In any of these embodiments, the method may further include determining a number of illumination changes from at least one of the plurality of pixels within a unit time and normalizing the number of changes by the determined frequency of the one or more frequencies associated with the at least one pixel to determine one or more amplitudes. For example, the unit time is about 1 ms or less. The time unit may be 1 ms or less.

[0041] In any of these embodiments, the method may further include projecting light onto an object within a field of view of the sensor, such as projecting uniform light or structured light. Uniform light may refer to light having a constant illumination, and structured light may refer to light having a structured illumination pattern. In such embodiments, the method may further include projecting light at an oblique angle relative to at least one surface of the object.

[0042] Additionally or alternatively, the method may further include eliminating timestamps associated with time differences equal to or greater than a time threshold. For example, the time threshold may be about 1 ms or greater. The time threshold may be 1 ms or greater.

[0043] According to a further aspect of the present disclosure, a non-transitory computer-readable medium is provided that includes instructions that, when executed by at least one processor, configure the at least one processor to perform the aforementioned method.

[0044] Additional objects and advantages of the disclosure will be set forth in part in the detailed description which follows, and in part will be obvious from the description, or may be learned by the practice of the disclosure. At least some of the objects and advantages of the disclosure will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims.

[0045] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments.

[0046] The accompanying drawings, which constitute a part of this specification, illustrate various embodiments and, together with the description, serve to explain the principles and features of the disclosed embodiments. [Brief description of the drawings]

[0047] [Figure 1] 1 is a schematic representation of an exemplary accelerometer-based vibration sensor known in the art. [Diagram 2] 1 is a schematic representation of an exemplary fiber optic vibration sensor known in the art. [Diagram 3] 1 is a schematic representation of a system using a pixel-based sensor as a vibration detector, according to an embodiment of the present disclosure. [Figure 4A] 1 is a schematic representation of an exemplary sensor pixel, according to an embodiment of the present disclosure. [Figure 4B] 1 is a schematic representation of an exemplary asynchronous sensor pixel, according to an embodiment of the present disclosure. [Figure 4C] 1 is a schematic representation of an exemplary sensor pixel, according to an embodiment of the present disclosure. [Figure 5A] 1 is a graphical representation of improving temporal resolution by using event-based sensors, according to an embodiment of the present disclosure. [Figure 5B] 1 is a graphical representation relating frequency to polarity information according to an embodiment of the present disclosure. [Figure 6] 1 is a flowchart of an exemplary method for detecting vibration frequency, according to an embodiment of the present disclosure. [Figure 7] 1 is a flowchart of an exemplary method for connecting events from sensors to clusters, according to an embodiment of the present disclosure. [Figure 8] 1 is a flowchart of an exemplary method for connecting events from sensors to clusters, according to an embodiment of the present disclosure. [Figure 9] 1 is a graphical illustration of an exemplary two-dimensional image of frequency and amplitude produced by an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0048] Embodiments of the present disclosure relate to systems and methods for detecting and analyzing vibrations. In some embodiments, event-based and / or asynchronous sensors may be used for vibration detection and imaging. Advantageously, exemplary embodiments can provide fast and accurate vibration sensing and imaging. Embodiments of the present disclosure may be implemented and used in a variety of applications and systems, such as analysis of moving machine elements such as rotating turbines or shafts, systems for monitoring engine or vehicle health, and other systems that benefit from accurate and efficient vibration analysis.

[0049] Although embodiments of the present disclosure are described with general reference to pixel-based sensors having photosensitive elements, it will be understood that the disclosed embodiments may be implemented in digital cameras, LIDAR, or other imaging systems. Additionally, while some embodiments are described in combination with a projector (such as a laser projector), it will be understood that embodiments of the present disclosure may be implemented with ambient lighting in combination with or in place of a projector. Also, some embodiments may use such components separate from the sensors and / or processors described herein.

[0050] Embodiments of the present disclosure may detect events to determine a vibration frequency at one or more pixels of an image sensor. For example, the events may include a polarity change. In addition, some embodiments may further determine an amplitude of the vibration at one or more pixels. As used herein, "polarity" may indicate whether a detected change in illuminance is associated with either an increase in illumination or a decrease in illumination.

[0051] "Frequency" as used herein may refer to any time characteristic that defines a vibration signature. For example, frequency may include the inverse of wavelength (e.g., the distance between peaks or between troughs), etc. Furthermore, if the vibration signature does not exhibit a single wavelength, the frequency may be an average or may include multiple components in Fourier space (also called "k-space"). Thus, "frequency" need not be singular, but may instead include a combined measurement of multiple measurements or multiple measurements themselves.

[0052] Figure 1 illustrates an accelerator-based detector 100. In the example of Figure 1, the vibrating device 101 may comprise a factory machine, a part of an automobile, or any other mechanical component or device that exhibits vibrations. To monitor these vibrations, an accelerometer 103 may be attached to a surface of the device 101, as shown in Figure 1.

[0053] Thus, the accelerometer 103 is attached to the device 101 and modifies any vibration signature of the device 101 accordingly. Compensating for any such modifications may require expensive post-processing. To avoid further confusion of the vibration signature, the processor 107 for using data from the accelerometer 103 may be implemented separately, rather than integrated with the accelerometer 103.

[0054] To communicate the results, the accelerometer 103 has at least one wire 105 connected to a processor 107 (or any other device that uses data from the accelerometer 103) to determine a vibration signature. However, the wire 105 may contact the device 101 (as shown in FIG. 1) and modify the vibration signature or characteristics of the device 101. Alternatively, the accelerometer 103 may communicate the results to the processor 107 using wireless communication 109 (e.g., radio frequency (RF) signals, etc.). However, the use of wireless communication may require a reduction in data density from the accelerometer 103, a memory attached to the accelerometer 103 to store the results before transmission (further modifying the vibration signature of the device 101), and / or non-continuous monitoring of the device 101 to enable transmission.

[0055] One improvement to a system such as 100 that relies on contact with an object that undergoes vibration is shown in FIG. 2. In FIG. 2, a schematic representation of a fiber optic vibration detector 200 is provided. In the example of FIG. 2, the vibrating device 201 may constitute a factory machine, a part of an automobile, or any other component or device that exhibits vibration. To monitor these vibrations, a non-coherent light source (e.g., a light emitting diode (LED) 203) may transmit (e.g., transmit 209a) electromagnetic radiation at a surface of the device 201. Although not shown in FIG. 2, in addition or as an alternative, a coherent light source (e.g., a laser light source, etc.) may transmit electromagnetic radiation at 209a. A photodetector 205 (e.g., one or more photodiodes, a charge-coupled detector (CCD), etc.) may detect a reflection (e.g., reflect 209b) of the transmitted radiation. As further shown in FIG. 2, one or more fiber optic cables (e.g., cables 207a and 207b) may guide the transmission and / or reflection to and / or from the surface of the device 200.

[0056] The photodetector 205 may detect vibrations based on changes in the intensity of the reflections (e.g., reflection 209) over time. However, the accuracy of this calculation may remain low. Furthermore, a single frequency is determined for the entire surface of the device 201 from which the transmission is reflected. Thus, vibrations with any kind of complexity (e.g., two or more frequencies on a single machine) cannot be adequately monitored using the system 200. In addition, multiple photodiodes are required to monitor the amplitude of the surface of the device 201, and more than one amplitude on a single machine still cannot be adequately monitored using the system 200.

[0057] Therefore, vibration monitoring techniques such as those shown in Figures 1 and 2 suffer from low accuracy. Moreover, these techniques suffer from lack of spatial resolution and other drawbacks. For example, the spatial resolution of the system 100 of Figure 1 cannot be improved without using multiple accelerometers that confound the vibration signature of the device 101. Furthermore, the system 200 of Figure 2 cannot distinguish (and often cannot even measure) the frequency and amplitude on the surface of the device 201. In addition, referring to Figure 1, changes in the vibration signature of the device 101 may indicate a problem with one or more accelerometers (e.g., accelerometer 103) rather than the device 101 itself.

[0058] Embodiments of the present disclosure provide greater precision and spatial resolution of detected frequency and amplitude. One exemplary embodiment is shown in Figure 3 and described further below.

[0059] 3 is a schematic representation of a system 300 for vibration detection, according to an embodiment of the present disclosure. As shown in FIG. 3, a projector 301 may transmit electromagnetic radiation (e.g., illumination 303) toward a surface of a vibrating device 305. By way of example, the vibrating device 305 may comprise a factory machine, a part of an automobile, or any other component or device that exhibits vibrations.

[0060] The projector 301 may comprise one or more laser generators, ultraviolet lamps, visible light lamps, or any other device configured to project electromagnetic radiation towards a surface. In some embodiments, the projector 301 may project structured light towards the surface of the vibration device 305. Within this context, structured light may refer to light that includes a particular illumination pattern. For example, the projector 301 may comprise a dot projector or any other structured light projector. Additionally or alternatively, the projector 301 may project uniform light towards the surface of the vibration device 305. Within this context, "uniform light" may refer to light having a constant intensity emitted from a light source having a substantially constant optical power output, such that for any stationary illuminated target area, the measured intensity of the incident light is substantially constant. For example, the projector 301 may comprise a light emitting diode (LED) or any other uniform light projector.

[0061] In some embodiments, as shown in Figure 3, the projector 301 may deliver illumination 303 perpendicular to at least a portion of a surface of the vibration device 305. Additionally or alternatively, although not shown in Figure 3, the projector 301 may deliver illumination 303 at an angle relative to at least a portion of a surface of the vibration device 305.

[0062] Although shown with a single projector 301, some embodiments may include multiple projectors. For example, multiple projectors may transmit electromagnetic radiation at different angles. In such embodiments, the vibrations and amplitudes may be associated with different depths as well as lengths and widths. Additionally, in some embodiments, vibrations in the lateral plane of the sensor 309 (as shown in FIG. 3) may be detected using different angles of projection and reflection.

[0063] In some embodiments, projector 301 may be configured to project multiple frequencies. In other embodiments, projector 301 may be configured to use a single frequency or a range of frequencies. Such embodiments may be used to distinguish reflections caused by patterns from noise in system 300. To this end, sensor 309 may be configured to use hardware-based filters, software-based filters, and / or optical filters to reduce or remove incident light that does not match (or is within a threshold frequency of) the frequency (or frequencies) projected by projector 301. For example, the frequency may be between 50 Hz and several kHz (e.g., 1 kHz, 2 kHz, 3 kHz, etc.).

[0064] While the embodiment of FIG. 3 is shown as using a projector 301, other embodiments may use ambient light in addition to or instead of the illumination 303 from the projector 301.

[0065] As shown in FIG. 3, the projected illumination 303 (and / or ambient light) may cause reflections 307 from the surface of the vibration device 305. The reflections may be captured by a sensor 309. In some embodiments, the sensor 309 may be an event-based sensor. As described below, the sensor 309 may include an array of pixels 400 (see FIG. 4A), an array of pixels 450 (see FIG. 4B), an array of pixels 470 (see FIG. 4C), or any other array of pixels coupled to a readout system. Each pixel of the sensor 309 may include at least one photosensitive element. The signal generated by the sensor 309 may be processed to calculate the frequency of the vibration device 305, as described further below. The frequency may be associated with a pixel of the sensor 309, and if the position of the projector 301 and the pixel are known, the system may map the frequency to a position in a two-dimensional projection of the surface of the vibration device 305. Other techniques described herein may be used to calculate a frequency map and / or amplitude depth associated with a position in a two-dimensional projection of the surface of the vibration device 305. Depth refers to the distance of the device 305 from the sensor 309, which may be calculated by triangulation or time-of-flight measurements.

[0066] 4A is a schematic representation of a sensor pixel 400, according to an embodiment of the present disclosure. The pixel 400 may be one of multiple pixels in an array of the sensor (e.g., a square, a circle, or any other regular or irregular shape formed by an array of pixels).

[0067] As used herein, a "pixel" refers to the smallest element of a sensor that outputs data based on light impinging on the pixel. In some embodiments, a pixel may be larger or contain more components than a conventional "pixel," as it may contain two or more photosensitive elements, other circuitry, etc., as shown, for example, in FIG. 4B, described below.

[0068] Although this disclosure refers to reflections caused by the projected pattern as received at a single pixel, the projected pattern may include a sufficient number of photons to cover and be received by multiple pixels, and therefore the triangulations described herein may be based on an average position of multiple pixels and / or may include multiple triangulations that include the position of each pixel within the multiple pixels.

[0069] As shown in FIG. 4A, the sensor pixel 400 includes a photosensitive element 401. The photosensitive element 401 may generate an electrical signal (e.g., voltage, current, etc.) based on the brightness of light impinging on the element 401. As used herein, a photosensitive element may include a photodiode (e.g., a pn junction or a PIN structure), or any other element configured to convert light into an electrical signal. The photodiode generates a current (e.g., l) that is proportional to the intensity of light impinging on the photodiode. ph ) may be generated.

[0070] As further shown in FIG. 4A, the sensor pixel 400 also includes a measurement circuit 405. The measurement circuit 405 may convert the current from the element 401 to an analog signal for readout. The measurement circuit 405 may be activated in response to an external control signal (e.g., an external clock cycle). Additionally or alternatively, the measurement circuit 405 may convert the signal from the element 401 to an analog signal, which is stored (e.g., in an on-chip and / or off-chip memory (not shown) accessed by the pixel 400) until an external control signal is received. The measurement circuit 405 may transmit the stored analog signal ("pixel data" in FIG. 4A) to a readout system in response to the external control signal.

[0071] Although not shown in FIG. 4A, a sensor using pixel 400 may include a row and column arbiter or other timing circuitry such that the array of pixels is triggered according to a clock cycle. Additionally, the timing circuitry may manage the transfer of analog signals to a readout system such that collisions are avoided, as described above. The readout system may convert the analog signals from the pixel array to digital signals for use in processing.

[0072] 4B is a schematic representation of a sensor pixel 450, according to an embodiment of the present disclosure. Pixel 450 may be one of multiple pixels in an array of the sensor (e.g., a square, a circle, or any other regular or irregular shape formed by an array of pixels).

[0073] As shown in FIG. 4B, the pixel 450 may include at least one photosensitive element 451. The photosensitive element 451 may generate an electrical signal based on the brightness of light impinging on the element 451. The pixel 450 may further include a state detector 455 (CD). As shown in the example of FIG. 4B, the detector 455 detects the state of the photosensitive element 451 (PD CD ) and is configured to generate a trigger signal (labeled "Trigger" in FIG. 4B) when an analog signal proportional to the brightness of light impinging on the photosensitive element 451 matches a condition. For example, the condition may include whether the analog signal exceeds a threshold (e.g., a voltage or current level). In such an example, the threshold may include a fixed threshold or a dynamic threshold. For example, the dynamic threshold may be set based on the fluctuation of the brightness (or the logarithm of the brightness), e.g., as a percentage of the most recent brightness level. Additionally or alternatively, one static or dynamic threshold may be used for positive fluctuations and another static or dynamic threshold may be used for negative fluctuations. The analog signal may include a voltage signal or a current signal.

[0074] As further shown in FIG. 4B, the pixel 450 may also include a photosensitive element 453 that generates an electrical signal based on the brightness of light impinging on the element 453. The pixel 450 may further include an exposure measurement circuit 457. As shown in FIG. 4B, the exposure measurement circuit 457 is connected to the photosensitive element 453 (PD EM ) may be configured to generate a measurement that is a function of the brightness of light impinging on the light sensitive element 453. The exposure measurement circuit 457 may generate the measurement in response to a trigger signal. Although shown in FIG. 4B as using an exposure measurement circuit 457, some embodiments may analyze the output signal from the photosensitive element 453 directly (e.g., using a control and readout system 459) and omit the exposure measurement circuit 457.

[0075] In some embodiments, the exposure measurement circuit may include an analog-to-digital converter. Examples of such embodiments are disclosed in U.S. Provisional Patent Application No. 62 / 690,948, entitled "Image Sensor with a Plurality of Super-Pixels," filed June 27, 2018, U.S. Provisional Patent Application No. 62 / 780,913, entitled "Image Sensor with a Plurality of Super-Pixels," filed December 17, 2018, and International Publication No. WO 2020 / 002562 A1, entitled "Image Sensor with a Plurality of Super Pixels," published January 2, 2020, which claims priority to the two aforementioned U.S. Provisional Patent Applications. The disclosures of these applications are expressly incorporated herein by reference. In such embodiments, the exposure measurement circuit 457 may reset the state detector 455 (e.g., using a "clear" signal, not shown in FIG. 4B) once the measurement is completed and / or transmitted to an external readout system.

[0076] In some embodiments, the exposure measurement circuit 457 may output measurements asynchronously to the control and readout system 459, for example, using an address-event expression (AER) communication protocol. In other embodiments, the readout from the exposure measurement circuit 457 may be clocked using an external control signal (e.g., labeled "CONTROL" in FIG. 4B). Additionally, as shown in FIG. 4B, in some embodiments, a trigger from the detector 455 may be output to the control and readout system 459, for example, using an asynchronous event readout (AER) communication protocol.

[0077] An example of the pixel 450 shown in Figure 4B is disclosed in U.S. Patent No. 8,780,240 and U.S. Patent No. 9,967,479, which are expressly incorporated herein by reference.

[0078] Although shown as distinct photosensitive elements, in some embodiments, photosensitive elements 451 and 453 may comprise a single element shared between state detector 455 and exposure measurement circuit 457. Examples of such embodiments are disclosed in European Patent Application No. 18170201.0, filed April 30, 2018, entitled "Systems and Methods for Asynchronous, Time-Based Image Sensing," and International Publication No. WO 2019 / 211317 A1, published November 7, 2019, which claims priority to the aforementioned European Patent Application. The disclosures of both of these applications are expressly incorporated herein by reference.

[0079] Additionally, although shown with one state detector and one exposure measurement circuit, some embodiments may include multiple exposure measurement circuits sharing a state detector such that multiple measurements are captured by a trigger signal. Examples of such embodiments are disclosed in U.S. Provisional Patent Application No. 62 / 690,948, entitled "Image Sensor with a Plurality of Super-Pixels," filed June 27, 2018, U.S. Provisional Patent Application No. 62 / 780,913, entitled "Image Sensor with a Plurality of Super-Pixels," filed December 17, 2018, and International Publication No. WO 2020 / 002562 A1, entitled "Image Sensor with a Plurality of Super Pixels," published January 2, 2020, which claims priority to both of the aforementioned U.S. Provisional Patent Applications. The disclosures of these applications are expressly incorporated herein by reference.

[0080] Although not shown in FIG. 4B, a sensor using pixel 450 may include row and column lines or other readout circuitry so that events generated by pixel 450 are read out of the sensor. Additionally, timing circuitry may manage the transfer of analog signals to a readout system such that collisions are avoided, as described above. The readout system may convert analog signals from the pixel array to digital signals for use in processing.

[0081] 4C is a schematic representation of a sensor pixel 470, according to an embodiment of the present disclosure. The pixel 470 may be one of multiple pixels in an array of the sensor (e.g., a square, a circle, or any other regular or irregular shape formed by an array of pixels).

[0082] As shown in FIG. 4C, the sensor pixel 470 includes a photosensitive element 473. The photosensitive element 473 may generate an electrical signal (e.g., a voltage, a current, etc.) based on the brightness of light impinging on the element 473. As disclosed herein, the photosensitive element may include a photodiode (e.g., a pn junction or a PIN structure), or any other element configured to convert light into an electrical signal. The photodiode may generate a current proportional to the intensity of light impinging on the photodiode.

[0083] As further shown in FIG. 4A, the sensor pixel 407 also includes a change detector 475. The change detector 475 may be configured to output a signal when a change in the detected illumination of the photosensitive element 473 exceeds a threshold. The change detector 475 may be activated in response to a control signal (e.g., a clock cycle or other control signal from a readout system). The change detector 475 may send an output signal ("Pixel Data" in FIG. 4C) to the readout system.

[0084] Although not shown in FIG. 4C, a sensor using pixel 470 may include a row and column arbiter or other timing circuitry such that the array of pixels is triggered according to a clock cycle. Additionally, the timing circuitry may manage the transfer of output signals to a readout system such that collisions are avoided, as described above. The readout system may convert the output signals from the pixel array (e.g., using an analog-to-digital converter) for use in processing.

[0085] In some embodiments, one or more change detectors may be shared by multiple photosensitive elements of the array of pixels. Clock cycles or other control signals may be provided (e.g., from a control and readout system) to coordinate the operation and output signals of the change detectors based on the detected illumination of the photosensitive elements.

[0086] In some embodiments, the change detector 475 provides an output signal to an external readout system (not shown in FIG. 4C) using an Address-Event Expression (AER) communication protocol.

[0087] FIG. 5A is a graphical representation of improving the temporal resolution by using an event-based sensor, according to an embodiment of the present disclosure. As explained above, an event-based sensor can provide a higher temporal resolution and therefore improve the accuracy of the detected vibration frequency. An example of this improved accuracy is shown in example 500 of FIG. 5A. The exposure rate 503 of a conventional camera pixel is shown, using shading, as a fixed frame rate, and the frequency of vibration at the object in the camera's field of view (which in example 500 of FIG. 5A is a sinusoid, but may include any other function). The y-axis represents intensity over a normalized scale ranging from 0 to 1. Thus, the sine wave in example 500 represents the temporal variation of light received by the camera due to vibration, and the length of each shaded box represents the integration time of a conventional camera pixel. Additionally, the height of each shaded box represents the resulting accumulation of impinging light during the integration time.

[0088] If the frequency exceeds half the exposure rate 503 (e.g., the Nyquist sampling rate of conventional pixels), the camera cannot reconstruct the signal based on the recorded frames. Furthermore, as shown in example 500, the exposure rate 503 can capture a large portion of the wavelengths in a single frame, further limiting the time resolution of the camera, thereby reducing the accuracy of any calculation of frequency therefrom.

[0089] The current solution is to increase the frame rate of conventional cameras, but this becomes impractical for many machines and objects that vibrate at high frequencies. Furthermore, as frame rates increase, greater illumination of the object in the camera's field of view is required to resolve signal fluctuations within shorter exposure times. However, this may be impractical depending on the illumination required and / or the environment in which the object is being used.

[0090] On the other hand, the event-based sensor of the present disclosure captures multiple events (e.g., labeled events 501a, 501b in FIG. 5A, etc.) at the same time that existing cameras can capture a single frame. Thus, the sensor's asynchronous pixels generate a pulse each time the signal amplitude exceeds a threshold relative to the amplitude in the previous pulse (e.g., as described above with respect to FIG. 4B). Thus, as shown in FIG. 5A, the vibration frequency can be determined with much greater accuracy based on the difference in time stamps between patterns of events that exhibit a wavy pattern or other similar vibration signature. Furthermore, higher frequencies that could not previously be tracked with conventional cameras can be tracked using the event-based sensor. Since enough information has been captured, a signal representative of the vibration can be reconstructed from the generated events.

[0091] FIG. 5B is a graphical representation relating frequency to polarity information. As shown in FIG. 5B, an event may have an associated polarity. For example, an event with a positive polarity may represent an increase in amplitude above a threshold (whether determined from the output of image sensor 400 in FIG. 4A or encoded in an event output from image sensor 450 in FIG. 4B). Furthermore, the change in polarity may be from positive to negative or from negative to positive. As shown in FIG. 5B, a wavelength may be calculated based on the same type of corresponding polarity changes (e.g., from one negative to positive change to a corresponding change or from one positive to negative change to a corresponding change). The determined event may then be used to calculate the frequency of the corresponding vibration represented by the event, as described below with respect to method 600 in FIG. 6.

[0092] 6 is a flow chart of an exemplary method 600 for detecting vibrations, according to an embodiment of the present disclosure. The method 600 of FIG. 6 may be performed using at least one processor, whether integrated as a processor on the same chip as a pixel-based sensor (e.g., a sensor having one or more of the pixels 400 of FIG. 4A, a sensor having one or more of the pixels 450 of FIG. 4B, a sensor having one or more of the pixels 470 of FIG. 4C, etc.) or provided separately as part of a processing system. The at least one processor may be in electrical communication with the sensor for purposes of transmitting and receiving signals, as further disclosed herein.

[0093] In step 601, the at least one processor may receive an output signal from a sensor. The sensor may comprise a plurality of pixels, each pixel including a photosensitive element, and may output a signal indicative of a change in illumination on each pixel (e.g., as described above with respect to FIGS. 4A-4C). For example, the at least one processor may receive a digital signal indicative of an event from the at least one sensor (e.g., using an on-chip bus connected to at least one transmitter configured to communicate with at least one memory, etc., via at least one network). Additionally or alternatively, the at least one processor may receive a digital signal encoding the amplitude of light impinging on a pixel of the sensor and track the amplitude over time to detect an event.

[0094] For example, the measurement circuit 405 (see FIG. 4A) may convert a signal from the photosensitive element 401 into an analog signal proportional to the brightness of the light impinging on the photosensitive element 401. The at least one processor may receive an analog signal from the measurement circuit 405 as one or more first signals, or may receive a digital signal based on the analog signal from an analog-to-digital converter in communication with the measurement circuit 405. Additionally or alternatively, as described above, the condition detector 455 (CD) may generate a trigger signal (e.g., “TRIGGER” in the example of FIG. 4B) when the first analog signal based on the light impinging on the photosensitive element 451 exceeds a predetermined threshold, and the exposure measurement circuit 457 may convert the signal from the photosensitive element 453 into a second analog signal proportional to the brightness of the light impinging on the photosensitive element 453 in response to the trigger signal. The at least one processor may receive a second analog signal from the exposure measurement circuit 457 as the one or more first signals, or may receive a digital signal based on the second analog signal from an analog-to-digital converter in communication with (or forming part of) the exposure measurement circuit 457. As yet another example, the change detector 475 (see FIG. 4C) may be configured to output a signal when a change in detected illuminance on the photosensitive element 473 exceeds a threshold. The at least one processor may be configured to receive the output signal from the change detector and determine a polarity based on the output signal, the polarity indicating whether the change in detected illuminance is increasing or decreasing.

[0095] At least one processor may receive the output signal multiple times. For example, in embodiments using synchronous sensors, the output signals may be captured at multiple clock cycles. As another example, in embodiments using asynchronous sensors, the output signals may be captured at different times. In some embodiments using asynchronous sensors, the readout may be clocked such that at least one processor receives some of the output signals on a different clock cycle than it receives other output signals.

[0096] In step 603, the at least one processor may detect one or more changes in illumination based on the output signal. The one or more changes may have associated polarity information. For example, the at least one processor may detect an event based on a change in amplitude between two of the output signals received from the sensor having a magnitude greater than one or more thresholds, etc., and then associate a polarity with the detected event. "Polarity" may refer to whether the detected change in illumination is associated with either an increase or decrease in illumination, which may be determined by a comparison of successive output signals. In embodiments using a sensor having one or more of the pixels (e.g., a sensor having one or more of the pixels 400 of FIG. 4A, a sensor having one or more of the pixels 450 of FIG. 4B, a sensor having one or more of the pixels 470 of FIG. 4C, etc.), the output signal may itself encode the event (as well as the associated polarity). In some embodiments, the output signal may be encoded using an address-event representation (AER) as shown in FIG. 4B, for example, but also applicable to the examples of FIG. 4A and FIG. 4C. In such an embodiment, each event may be encoded as a tuple (x,y,t,p), where x and y are the two-dimensional coordinates of the pixel from which the output signal was emitted, t is the timestamp of when the pixel captured the event (or when a readout system (such as system 459) converted the event into a digital output), and p is the polarity of the event (e.g., zero (0) for negative polarity, such as a decrease in amplitude, and (1) for positive polarity, such as an increase in amplitude). Thus, the at least one processor can detect the polarity information by determining the polarity associated with the event in the output signal.

[0097] In some embodiments, the at least one processor may be configured to detect a positive polarity from the sensor when the change in illumination is an increasing change and a negative polarity from the sensor when the change in illumination is a decreasing change. Further, these polarities may be associated with a set of pixels of the plurality of pixels of the sensor. Thus, the at least one processor may determine one or more frequencies (described below in step 611) associated with the set of pixels based on the polarities. For example, the determined frequencies may be mapped to pixel locations such that an image of the determined frequencies in the two-dimensional plane of the sensor may be formulated.

[0098] In some embodiments, the at least one processor can map the determined frequency based on an address encoded in the output signal by the sensor. For example, the sensor (or a readout system in communication with the sensor) may encode an address of a pixel from which the output signal originates. Thus, the at least one processor can associate the output signal with a particular pixel of the sensor based on the address encoded in the output signal.

[0099] At step 605, the at least one processor may determine a timestamp associated with one or more changes in the illumination. The timestamp may encode a global time (e.g., Greenwich Mean Time (GMT), Coordinated Universal Time (UTC), etc.), a system time (e.g., for a similar readout system, etc.), or another indicator of the time distance between the output signals.

[0100] In some embodiments, the at least one processor can determine the timestamps by extracting multiple timestamps from a readout system (e.g., control and readout 459) connected to or otherwise associated with the sensor. For example, the readout system may encode one or more timestamps based on an Address-Event Representation (AER) protocol. In addition, as described above, the readout system may be further configured to encode an address that identifies an associated pixel for each measurement.

[0101] In step 607, the at least one processor may analyze the timestamps and polarity information to detect vibration of an object within the field of view of the sensor. In one example, the at least one processor may use the multiple timestamps to determine a time interval between one or more polarity changes. Such a time interval may represent a wavelength (or other measure of the period of a function representing the vibration) of the vibrating object (e.g., the vibrating device 305 of FIG. 3). The at least one processor may further calculate a combination value associated with the determined time interval. For example, the combination value may include an average, median, or other statistical combination of the time intervals. The at least one processor may further determine one or more frequencies of the vibration based on the inverse of the combination value. In an embodiment in which the time interval represents a fraction (or multiple) of a wavelength or other measure of periodicity, one or more vibration frequencies may be determined based on the frequency adjusted for the fraction (or multiple). For example, the time interval between one or more different polarity changes (e.g., from positive to negative to negative to positive, or from negative to positive to positive) may represent a portion of a wavelength (e.g., from peak to trough, or from trough to peak) and therefore may be adjusted by a corresponding multiple to calculate the wavelength.

[0102] In some embodiments, the at least one processor may further determine a deviation value associated with the time interval. For example, the deviation value may include a standard deviation or other measure of error in the time interval. Additionally or alternatively, the at least one processor may determine a frequency associated with each time interval and then determine a deviation value associated with the frequency. In such embodiments, the at least one processor may validate one or more frequencies based on when the deviation falls below a validation threshold. For example, the validation threshold may be about 1%-10% of the mean or median. If the deviation does not fall below the threshold, the at least one processor may output a zero frequency, output an error message, throw an exception, or otherwise indicate that the time interval did not exhibit a consistent frequency.

[0103] In some embodiments, the at least one processor may first cluster the time intervals before performing step 607. For example, the at least one processor may cluster the time intervals based on size. Thus, the at least one processor may use the calculations described above to determine a primary frequency associated with a cluster having a maximum number of time intervals and one or more secondary frequencies associated with clusters having a fewer number of time intervals.

[0104] In step 609, the at least one processor may determine whether the analyzed timestamps meet one or more stability criteria. For example, as described above with respect to step 607, a validation threshold may filter out time intervals that do not exhibit a consistent frequency. Additionally or alternatively, time intervals that are themselves too short or too long may be filtered out as anomalous. Additionally or alternatively, the stability criteria may include one or more of a periodicity measure, a variation in or rate of change of a frequency determined based on the timestamp, a movement of the timestamp across pixels, a variation in the number of detected events within each period of the frequency, and the like. Furthermore, when the analyzed timestamps do not reflect a stable frequency, the at least one processor may output a zero frequency, output an error message, throw an exception, or otherwise indicate that the time interval did not exhibit a consistent frequency. However, some embodiments may omit step 609 entirely.

[0105] In step 611, the at least one processor may determine one or more frequencies of the detected vibration. For example, as described above with respect to step 607, the one or more frequencies may include the inverse of the combined values ​​of the time intervals. In another example, the at least one processor may identify a series of events from the output signal that indicate increasing or decreasing portions of the vibration signature. The at least one processor may then map the events to points on the predicted vibration signature and identify points that are expected to be one wavelength (or a fraction of a wavelength, or a multiple of a wavelength). The at least one processor may use the inverse of the time intervals between the identified points to determine the one or more frequencies.

[0106] In some embodiments, the method 600 may include additional steps. For example, the at least one processor may filter the illumination changes to remove spurious events. For example, as described above, the at least one processor may eliminate time intervals that are too long and / or too short. In such an example, the at least one processor may eliminate timestamps that have a difference that is a time interval apart that is greater than the time threshold, and / or eliminate such time intervals that are greater than the time threshold. Thus, the at least one processor may delete (or otherwise ignore) the timestamp itself, or ignore time intervals that exceed the time threshold while still using the timestamp to calculate time intervals within the time threshold relative to other timestamps. For example, the time threshold may be approximately 1 ms or greater. Thus, any corresponding events that have a time interval of 1 ms or greater are too far away to represent the wavelength (or a fraction or multiple thereof) of the vibration signature. The time threshold may be adjusted according to the range of frequencies expected to be measured by the system.

[0107] In some embodiments not shown in FIG. 6, the method 600 may be recursive. For example, the at least one processor may repeat steps 601, 603, 605, 607, and 609 with each new set of output signals from the sensor (e.g., generated and / or received at the next clock cycle). The new output signals may generate a calculation of a new frequency and / or a comparison of one or more frequencies associated with the new output signals to one or more frequencies calculated in a previous iteration of step 611. Thus, the at least one processor may detect any changes from the previous one or more frequencies and the new frequency and generate a warning or other error message if the change is unexpected. Additionally or alternatively, the at least one processor may determine whether any changes from the previous one or more frequencies and the new frequency are associated with an expected event (e.g., power on, power off, etc.) and generate a message accordingly.

[0108] In some embodiments, the at least one processor may further determine a depth associated with one or more frequencies. For example, in addition to mapping the frequency to a particular pixel that generated the polarity change used to determine the frequency, the at least one processor may use a time-of-flight algorithm, triangulation, or other calculation to determine a depth associated with a pixel based on a known location from which the transmitted light was emitted (e.g., a location associated with projector 301 in FIG. 3). Thus, a multi-dimensional mapping of frequencies may be generated.

[0109] Although the method 600 determines the frequency, embodiments of the present disclosure may also determine the amplitude. Thus, the at least one processor may further analyze measurements associated with one or more of the plurality of pixels to determine one or more amplitudes of the vibration. For example, the at least one processor may apply a spatial filter (e.g., a Gaussian filter, etc.) to the measurements from the plurality of pixels and determine one or more amplitudes based on the applied spatial filter. In embodiments in which the shape of the vibrating object is known (e.g., stored in a database accessible by the at least one processor, received by the at least one processor via a port such as a network port or peripheral port, etc.), the at least one processor may perform two-dimensional shape tracking on the measurements to retrieve the shape of the object undergoing the vibration and determine one or more amplitudes. For example, the two-dimensional shape tracking may include applying an iterative closest point algorithm.

[0110] Additionally or alternatively, FIG. 7 shows an example method 700 for determining the amplitude of vibration based on frequency detected, e.g., using method 600, applied to a signal from a pixel-based sensor, e.g., a sensor having one or more of pixels 400 of FIG. 4A, a sensor 450 having one or more of pixels 450 of FIG. 4B, etc.

[0111] As mentioned above, Figure 7 illustrates a flowchart of an exemplary method 700 for determining an amplitude of vibration using a detected frequency, according to an embodiment of the present disclosure. The method 700 of Figure 7 may be performed using at least one processor, whether integrated as a processor on the same chip as a sensor (e.g., a sensor having one or more of the pixels 400 of Figure 4A, a sensor having one or more of the pixels 450 of Figure 4B, a sensor having one or more of the pixels 470 of Figure 4C, etc.) or provided separately as part of a processing system. The at least one processor may be in electrical communication with the sensor for purposes of transmitting and receiving signals, as further disclosed herein.

[0112] In step 701, at least one processor may receive or determine an event from a sensor (e.g., a sensor having one or more of pixels 400 of FIG. 4A, a sensor having one or more of pixels 450 of FIG. 4B, a sensor having one or more of pixels 470 of FIG. 4C, etc.). As described above with respect to step 603 of method 600, the event may include an output signal from an event-based sensor, or an event extracted from an output signal of a continuous sensor (e.g., using a clock circuit).

[0113] In step 703, the at least one processor may determine a number of illumination changes from at least one of the plurality of pixels in a unit time. In some embodiments, the time unit may be about 1 ms or less. The illumination changes may be encoded as events. For example, the at least one processor may receive or decode events from a sensor (e.g., a sensor having one or more of the pixels 400 of FIG. 4A, a sensor having one or more of the pixels 450 of FIG. 4B, a sensor having one or more of the pixels 470 of FIG. 4C, etc.). As described above with respect to step 603 of method 600, the events may include signals from an event-based sensor or events extracted from a continuous sensor's signal (e.g., using a clock circuit).

[0114] In step 705, the at least one processor may determine one of the one or more frequencies associated with the at least one pixel. For example, the frequency may be determined using method 600 or a similar calculation that converts events (or polarity changes) to frequencies.

[0115] In step 707, the at least one processor may normalize the number of changes by the determined frequency. For example, the at least one processor may divide the number of changes by the determined frequency.

[0116] In step 709, the at least one processor may use normalization to determine one or more amplitudes associated with the determined frequencies. For example, the amplitudes may include the output of step 707 or may include the output of step 707 scaled, for example, according to a factor based on a known shape of the object undergoing vibration, etc.

[0117] 8 is a flow chart of an exemplary method 800 for detecting an event burst, according to an embodiment of the present disclosure. The method 800 of FIG. 8 may be performed using at least one processor, whether integrated as a processor on the same chip as a sensor (e.g., a sensor having one or more of the pixels 400 of FIG. 4A, a sensor having one or more of the pixels 450 of FIG. 4B, a sensor having one or more of the pixels 470 of FIG. 4C, etc.) or provided separately as part of a processing system. The at least one processor may be in electrical communication with the sensor for purposes of transmitting and receiving signals, as further disclosed herein.

[0118] In step 801, the at least one processor may receive or determine an event from a sensor (e.g., a sensor having one or more of pixels 400 of FIG. 4A, a sensor having one or more of pixels 450 of FIG. 4B, a sensor having one or more of pixels 470 of FIG. 4C, etc.). As described above with respect to step 603 of method 600, the event may include an output signal from an event-based sensor, or an event determined from an output signal of a continuous sensor (e.g., using a clock circuit).

[0119] In step 803, the at least one processor may determine or verify the polarity of the event. For example, the at least one processor may determine whether the polarity matches the expected polarity for the event, is the same as the previous event if multiple increases or decreases are expected, or is different from the previous event if a polarity change is expected. For example, the projection pattern may be configured to always generate multiple (e.g., 2, 3, etc.) events to signal an increase or decrease signal. Such multiple may allow for filtering of noise in step 803. If the polarity is not valid, the at least one processor may discard the event and start over with a new event in step 801. Additionally or alternatively, if the polarity is not valid, the at least one processor may discard the current burst and use the event from step 801 as the start of a new potential burst.

[0120] In step 805, the at least one processor may discard a received event if it is too far away in time from a previous event (e.g., if the time difference exceeds a threshold). Thus, the at least one processor may avoid connecting events that are too far apart in time to form part of a single burst. If the events are too far apart, the at least one processor may discard the event and start over with a new event in step 801. Additionally or alternatively, if the events are too far apart, the at least one processor may discard the current burst and use the event from step 801 as the start of a new potential burst.

[0121] In step 807, the at least one processor may increment an event counter for an associated pixel. For example, the associated pixel may include the pixel at which the event of step 801 was received. The event counter may include an integer that counts events received in the recursive execution of step 801 that are considered to be in the same burst under steps 803 and 805.

[0122] In step 809, the at least one processor may extract a burst if the event counter exceeds an event threshold. For example, the event threshold may include 2 to 10 events. In other embodiments, a larger event threshold may be used. If a burst is extracted, the at least one processor may reset the event counter. If the event counter does not exceed the event threshold, the at least one processor may return to step 801 without resetting the event counter. Thus, under steps 803 and 805, additional events that qualify as being within the same burst may be detected and added to the event counter in step 807.

[0123] In some embodiments, method 800 may further include discarding a received event if it is too far in time from the first event of the current burst, thus preventing noise from inadvertently extending a burst beyond a threshold.

[0124] Additionally or alternatively, method 800 may track several events per region such that bursts are only detected within a region, rather than in a single pixel or across the entire sensor. Thus, method 800 may allow for detection of simultaneous bursts on different portions of the sensor.

[0125] Whenever an event is discarded, the at least one processor may reset the event counter. Alternatively, in some embodiments, the at least one processor may store the corresponding event counter even when the event is discarded. Some embodiments may use a combination of storing and discarding. For example, the event counter may be saved if the event is discarded in step 803, but reset if the event is discarded in step 805.

[0126] By detecting bursts rather than individual events, the at least one processor may measure the time interval between polarity changes with greater accuracy. For example, the at least one processor may use method 800 to eliminate spurious events that represent noise or other imperfections in the measurement that reduce the accuracy of the calculated frequency (and therefore, amplitude). Exemplary embodiments and features for implementing method 800 are described in International Patent Application No. PCT / EP2019 / 051919, filed January 30, 2019, entitled "Method and Apparatus of Processing a Signal from an Event-Based Sensor," and published as International Publication No. WO 2019 / 145516 A1 on August 1, 2019. The disclosures of this application and its publications are expressly incorporated herein by reference.

[0127] In addition to or in lieu of method 800, the at least one processor may cluster events before analyzing the clusters for frequency. For example, the at least one processor may cluster events for which at least one connectivity or other criterion is met. For example, the at least one processor may determine a time distance between a new event and one or more recent events and connect them if the time distance meets a threshold. Additionally or alternatively, the at least one processor may determine a spatial distance between a new event and one or more recent events and connect them if the spatial distance meets a threshold. Thus, the at least one connection criterion may include a temporal threshold, a spatial threshold, or any combination thereof. In one example combination, the spatial threshold may be adjusted based on which of multiple time thresholds are met. In such an example, events closer in time may be expected to be closer in space. In another example combination, the time threshold may be adjusted based on which of multiple spatial thresholds are met. In such an example, events closer in space may be expected to be closer in time. An exemplary clustering algorithm is described in European Patent Application No. 19154401.4, entitled "Method of Processing Information from an Event-Based Sensor," filed on January 30, 2019, the disclosure of which is expressly incorporated herein by reference.

[0128] By clustering events over time and discarding (or otherwise ignoring) unclustered events, the at least one processor can eliminate spurious time intervals between polarity changes that represent noise or other imperfections in the measurement that reduce the accuracy of the calculated frequency (and therefore, amplitude). By clustering events over space and discarding (or otherwise ignoring) unclustered events, the at least one processor can track the vibration pattern exhibited by the surface of an object within the field of view of the pixel across pixels (such as, for example, vibrating device 305 of FIG. 3) (e.g., if the amplitude of the vibration is large enough such that at least some reflections caused by the vibrating object (or component) are received by a different pixel than other reflections caused by the vibrating object (or component)).

[0129] 9 shows an example 900 of frequency detection using polar mapping (as described above) and clustering before determining amplitude from the output of a Sisley® camera. A depicted object (e.g., a motor) may be detected early by an unexpected shift in frequency or amplitude, and such shift may be detected in the two-dimensional image, but not in the overall average of the entire surface. Example 900 includes a two-dimensional image of the detected frequency and a similar two-dimensional image of the detected amplitude.

[0130] Although sensor embodiments have been described herein with reference to examples in Figures 4A-4C, any form of sensor adapted to capture a signal based on the brightness of light impinging on one or more photosensitive elements (e.g., pixels) may be used. Thus, any combination of transistors, capacitors, switches, and / or other circuit components configured to perform such capture may be used in the systems of the present disclosure. Additionally, embodiments of the present disclosure may use any synchronous sensor (e.g., a sensor having one or more of pixels 400 of Figure 4A), or any event-based sensor (e.g., a sensor having one or more of pixels 450 of Figure 4B), or any variation thereof (e.g., a sensor having one or more of pixels 470 of Figure 4C).

[0131] The above description has been presented for illustrative purposes. It is not exhaustive and is not limited to the precise forms or embodiments disclosed. Modifications and adaptations of the embodiments will become apparent from consideration of the specification and practice of the disclosed embodiments. For example, while the implementation described includes hardware, the systems and methods according to the present disclosure can be implemented using hardware and software. In addition, while certain components have been described as being coupled together, such components may be integrated with each other or distributed in any suitable manner.

[0132] Furthermore, although exemplary embodiments have been described herein, the scope includes any and all embodiments having equivalent elements, modifications, omissions, combinations (e.g., of aspects across various embodiments), adaptations, and / or alterations based on this disclosure. Elements in the claims are to be interpreted broadly based on the language used in the claims and are not limited to the examples described herein or in the filing of this application, which examples are to be interpreted as non-exclusive. Furthermore, the steps of the disclosed methods can be modified in any manner, including rearranging steps and / or inserting or deleting steps.

[0133] The features and advantages of the present disclosure are apparent from the detailed specification, and therefore, the appended claims are intended to cover all systems and methods within the true spirit and scope of the present disclosure. As used herein, the indefinite articles "a" and "an" mean "one or more." Similarly, the use of a plural term does not necessarily mean a plural, unless otherwise clear in a given context. Terms such as "and" or "or" mean "and / or" unless otherwise indicated. Furthermore, since numerous modifications and variations will readily occur upon study of the present disclosure, it is not desired to limit the present disclosure to the exact construction and operation shown and described, and therefore, all suitable modifications and equivalents may be reclassified as falling within the scope of the present disclosure.

[0134] Other embodiments will be apparent from consideration of the specification and practice of the embodiments disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosed embodiments being indicated by the following claims.

Claims

1. 1. A system for detecting vibrations, comprising: a plurality of photosensitive elements configured to detect an intensity of illumination on the elements; a plurality of change detectors configured to output a signal when a change in the detected illuminance on one or more of the photosensitive elements exceeds a threshold; A sensor comprising: receiving output signals from the plurality of change detectors; determining a polarity based on the output signal, the polarity indicating whether the illumination is increasing or decreasing; determining a timestamp associated with the output signals from the plurality of change detectors; analyzing the timestamps and the polarity to detect vibrations of an object within a field of view of the sensor by using the timestamps to calculate a time interval between changes in polarity; eliminating time intervals that are separated by a time interval greater than a time interval threshold; determining one or more frequencies of the detected vibration based on the remaining calculated time interval; At least one processor configured to A system comprising:

2. 1. A system for detecting vibrations, comprising: a plurality of photosensitive elements configured to detect an intensity of illumination on the elements; a plurality of change detectors configured to output a signal when a change in the detected illuminance on one or more of the photosensitive elements exceeds a threshold; A sensor comprising: receiving output signals from the plurality of change detectors; determining a polarity based on the output signal, the polarity indicating whether the illumination is increasing or decreasing; determining a timestamp associated with the output signals from the plurality of change detectors; analyzing the timestamps and the polarity to detect vibrations of an object within a field of view of the sensor by using the timestamps to calculate a time interval between changes in polarity; eliminating time intervals that are separated by a time interval less than or equal to a time interval threshold; determining one or more frequencies of the detected vibration based on the remaining calculated time interval; At least one processor configured to A system comprising:

3. 3. The system of claim 1 or 2, wherein the time interval threshold is selected according to the range of frequencies expected to be measured by the system.

4. 3. The system of claim 1 or 2, wherein a positive polarity is associated with increased illumination and a negative polarity is associated with decreased illumination.

5. the polarity is associated with a set of pixels, each pixel including at least one of the plurality of photosensitive elements; and optionally the one or more frequencies are associated with the set of pixels based on the polarity; A system according to any one of claims 1 to 4.

6. 6. The system of claim 1 , wherein the at least one processor is configured to determine the timestamps associated with the output signals from the plurality of change detectors by extracting a plurality of timestamps from a readout system associated with the sensor.

7. The system of claim 6 , wherein the read system encodes the multiple timestamps based on an Address-Event Representation (AER) protocol.

8. 8. A system according to claim 6 or 7, wherein the readout system encodes for each output signal an address which identifies the associated pixel.

9. 9. The system of claim 1, wherein the at least one processor is configured to calculate a time interval between one or more polarity changes using a plurality of timestamps and to determine the one or more frequencies based on the calculated time intervals.

10. The at least one processor Calculating a mean or median and a deviation associated with said calculated time interval; determining the one or more frequencies based on an inverse of the average value when the average value is calculated, or determining the one or more frequencies based on an inverse of the median value when the median value is calculated; verifying the one or more frequencies when the deviation is below a verification threshold; The system of claim 9 , further configured to:

11. The at least one processor is configured to analyze the output signal to determine one or more amplitudes of the vibrations, and optionally the at least one processor is configured to apply a spatial filter to the output signal and determine the one or more amplitudes based on the applied spatial filter. A system according to any one of claims 1 to 10.

12. The at least one processor is configured to perform two-dimensional shape tracing on the output signals to determine a shape of the vibrated object and to determine the one or more amplitudes; and optionally the at least one processor is configured to perform the two-dimensional shape tracking by applying an iterative closest point algorithm. The system of claim 11.

13. 13. The system of claim 11 or 12, wherein the at least one processor is configured to determine a number of changes in illuminance from at least one of a plurality of pixels within a unit time and normalize the number of changes by a determined frequency of the one or more frequencies associated with at least one of the plurality of pixels to determine the one or more amplitudes.

14. and an external light source configured to project light onto the object within the field of view of the sensor. A system according to any one of claims 1 to 13.

15. The system of claim 14, wherein the external light source projects light having a constant illuminance.

16. The system of claim 14, wherein the external light source projects light including a structured illumination pattern.

17. 17. The system of claim 15 or 16, wherein the external light source projects the light at an oblique angle to at least one surface of the object.

18. 1. A method for detecting vibrations, comprising: receiving, from a sensor including a plurality of pixels, an output signal associated with an illuminance on one or more of the pixels, each of the pixels including at least one photosensitive element; determining a polarity based on the output signal, the polarity indicating whether the illumination on the one or more pixels is increasing or decreasing; determining a timestamp associated with the output signal; using a plurality of said timestamps to calculate a time interval between one or more polarity changes to detect vibration of an object within a field of view of said sensor; eliminating time intervals that are separated by a time interval greater than a time interval threshold; determining one or more frequencies of the detected vibration based on the remaining calculated time interval; The method includes:

19. 1. A method for detecting vibrations, comprising: receiving, from a sensor including a plurality of pixels, an output signal associated with an illuminance on one or more of the pixels, each of the pixels including at least one photosensitive element; determining a polarity based on the output signal, the polarity indicating whether the illumination on the one or more pixels is increasing or decreasing; determining a timestamp associated with the output signal; using a plurality of said timestamps to calculate a time interval between one or more polarity changes to detect vibration of an object within a field of view of said sensor; eliminating time intervals that are separated by a time interval less than or equal to a time interval threshold; determining one or more frequencies of the detected vibration based on the remaining calculated time interval; The method includes:

20. 20. A non-transitory computer readable medium comprising instructions that, when executed by at least one processor, configure the at least one processor to perform the method of claim 18 or 19.

Images