Vibration Sensing Method and System Based on Event Camera

Abstract

The present invention provides a vibration sensing method and system based on an event camera, which can conveniently and efficiently acquire and reconstruct vibration signals. The vibration sensing method based on an event camera includes: Step 1, using a laser emitter to irradiate a position with relatively low reflectivity of the vibrating object to be measured, and aligning the event camera with the laser spot on the object to be measured; Step 2, adjusting the lens of the event camera so that it focuses on a certain plane between the object to be measured and the event camera, and capturing a laser speckle event stream; Step 3, the laser speckle patterns are concentrated in a certain area of the event camera's screen. For isolated event points, they are regarded as hardware inherent noise and removed to obtain a denoised event stream; Step 4, segmenting the event stream according to the required sampling period based on timestamps to obtain a segmented event stream; Step 5, counting the number of positive event points and negative event points in each segment of the event stream, and subtracting the number of negative event points from the number of positive event points to obtain the reconstructed vibration signal.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a vibration perception method and system based on an event camera. Background Art

[0002] Vibration is a ubiquitous phenomenon, and the vibration principle is widely applied in industries such as music, architecture, medical treatment, manufacturing, building materials, detection, and military. There are many fine branches, and in-depth research on any branch can promote the progress of science and drive social progress. Therefore, exploring a more efficient and accurate vibration perception technology is of great significance for the continuous development of various disciplines.

[0003] Currently, the common vibration perception methods are roughly divided into three types. One is manual measurement, the second is sensor-based vibration perception, and the third is vision-based vibration perception. Manual inspection and measurement require a large amount of manpower, and it is difficult to guarantee the measurement accuracy and it is affected by subjective factors. The sensor-based vibration monitoring method can grasp the vibration state of the monitoring target in real time through sensors, but sensors need to be installed on each object to be measured, consuming a large amount of material and financial resources. At the same time, the accuracy of the sensors is affected by environmental factors such as temperature and radiation. The vision-based vibration perception method can quickly perceive the vibration parameters of the target only through the picture, but it needs to use an expensive high-speed camera to capture the picture, and then use the digital image correlation method, the marker tracking method or the optical flow method to analyze the vibration of the object. It is necessary to preset markers on the object to be measured, the preparation work is cumbersome, and it is sensitive to light and is easily affected by the environment, resulting in noise. Summary of the Invention

[0004] The present invention is made to solve the above problems, and aims to provide a vibration perception method and system based on an event camera, which can conveniently and efficiently acquire and reconstruct vibration signals, with simple operation and high accuracy.

[0005] To achieve the above object, the present invention adopts the following solutions:

[0006] <Method>

[0007] The present invention provides a vibration perception method based on an event camera, which is characterized by including the following steps:

[0008] Step 1, in a static scene, use a laser emitter to irradiate a position with a lower reflectivity of the vibrating object to be measured, and aim the event camera at the laser spot on the object to be measured;

[0009] Step 2, use the event camera to capture the laser speckle motion image; adjust the event camera lens to focus on a certain plane between the object to be measured and the event camera, and capture the laser speckle event stream

[0010] Step 3: The laser speckle pattern is concentrated in a certain area of ​​the event camera screen. The isolated event points are treated as hardware inherent noise and removed to obtain the denoised event stream.

[0011] Step 4: Segment the event stream by timestamp according to the required sampling period to obtain the segmented event stream

[0012] Step 5, vibration signal reconstruction;

[0013] Count each event stream ε n Number of Zhongzheng event points and the number of negative event points Subtract the number of positive incident points from the number of negative incident points Get the reconstructed vibration signal

[0014] Preferably, the vibration sensing method based on event camera provided by the present invention may further include: step 6, setting a noise signal threshold value, The signal below the threshold is recorded as noise removal, and the denoised vibration signal is obtained.

[0015] Step 7: Based on the frequency distribution characteristics of the measured signal, design the filter group e(n) to strengthen or weaken the energy of a specific frequency band, and obtain the final vibration signal after equalization.

[0016] Preferably, the vibration sensing method based on event camera provided by the present invention may also have the following feature: in step 1, the laser point irradiated by the laser transmitter on the object to be measured and the event camera should be on the same optical axis as much as possible.

[0017] Preferably, in the vibration sensing method based on event camera provided by the present invention, in step 4, the segmentation time window length Δt=200us and the time window overlap amount ο=0us are taken.

[0018] Specifically, in the vibration sensing method based on event camera provided by the present invention, in step 5, the vibration reconstruction principle is:

[0019] In the nth event stream, for each pixel x of the event camera i :

[0020]

[0021] In one image:

[0022]

[0023] Where N ris the resolution of the event camera; let represent the geometric mean brightness of the image obtained at time t. On the right side of the equation, let:

[0024]

[0025]

[0026]

[0027] respectively represent the statistical values of the number of positive and negative event points in the nth event stream. After simplification, we can get:

[0028]

[0029] That is, when the average brightness changes periodically, the left side of the equation will also change with the same period. Then also changes with the same period; therefore, by using the obtained signal contains the frequency information of the audio vibration signal of the sound source, and the reconstructed audio vibration signal is obtained:

[0030]

[0031] Preferably, in the vibration perception method based on an event camera provided by the present invention, in step 6, a noise signal threshold is set to make all values with an amplitude less than η thresh equal to zero:

[0032]

[0033] <System>

[0034] Furthermore, the present invention also provides a vibration perception system based on an event camera, which can automatically implement the above <method>. It is characterized in that it includes:

[0035] An operation prompt unit that prompts the operator to irradiate a position with a lower reflectivity of the vibrating object to be measured using a laser emitter in a static scene, and align the event camera with the laser spot on the object to be measured; and prompts the operator to adjust the lens of the event camera to focus on a certain plane between the object to be measured and the event camera for shooting;

[0036] An event stream acquisition unit that uses an event camera to capture the laser speckle motion image and obtains the laser speckle event stream

[0037] An event stream denoising unit. Since the laser speckle pattern is concentrated in a certain area of the event camera screen, isolated event points are regarded as the inherent noise of the hardware and removed to obtain the denoised event stream

[0038] The time-domain sampling unit divides the event stream according to the time stamp at the required sampling period to obtain the segmented event stream.

[0039] The vibration signal reconstruction unit counts each segment of the event stream ε n The number of positive event points And the number of negative event points Subtract the number of positive event points from the number of negative event points To obtain the reconstructed vibration signal

[0040] The control unit is communicatively connected to the operation prompt unit, the event stream acquisition unit, the event stream denoising unit, the time-domain sampling unit, and the vibration signal reconstruction unit to control their operations.

[0041] Preferably, the vibration perception system based on an event camera provided by the present invention may further include: a signal denoising unit communicatively connected to the control unit. Based on the noise signal threshold, the signals below the threshold in Are marked as noise removed to obtain the denoised vibration signal And a signal equalization unit communicatively connected to the control unit. According to the frequency distribution characteristics of the measured signal, a filter bank e(n) is designed to strengthen or weaken the energy of a specific frequency band, and the final vibration signal is obtained after equalization.

[0042] Preferably, the vibration perception system based on an event camera provided by the present invention may further include: an input display unit communicatively connected to the control unit, which is used for the operator to input operation instructions and perform corresponding displays. For example, the input display unit can display the data obtained by each unit, the processing process, and the output results in the form of graphs or tables respectively for the operator to view.

[0043] Specifically, in the vibration signal reconstruction unit of the vibration perception system based on an event camera provided by the present invention, the vibration reconstruction principle is:

[0044] In the nth segment of the event stream, for each pixel point x of the event camera i :

[0045]

[0046] In an image:

[0047]

[0048] Where N r Is the resolution of the event camera; let Represents the geometric mean brightness of the image obtained at time t, and the right side of the equation is:

[0049]

[0050]

[0051]

[0052] Respectively represent the statistical values of the number of positive and negative event points in the nth segment of the event stream. After simplification, we can get:

[0053]

[0054] That is, when the average brightness changes periodically, the left side of the equation will also change with the same period. Then also changes with the same period; Therefore, by using The obtained signal contains the frequency information of the audio vibration signal of the sound source, and the reconstructed audio vibration signal is obtained:

[0055]

[0056] Functions and effects of the invention

[0057] The vibration perception method and system based on an event camera provided by the present invention uses a laser emitter to irradiate the vibrating object to be measured, and uses the laser speckle pattern captured by the event camera in the out-of-focus state to magnify the object vibration. By using the principle of frequency consistency between the speckle average brightness and the object vibration, time-domain sampling is performed on the event stream obtained from the speckle movement captured by the event camera, and the change frequency of the speckle average brightness is measured to reconstruct and restore the vibration signal, realizing long-distance visual vibration measurement (remote capture of vibration), effectively solving the problems of low accuracy, complex operation, and high consumption of human and material resources of other methods. Description of the drawings

[0058] Figure 1 It is a structural diagram of the hardware platform related to the first embodiment of the present invention;

[0059] Figure 2 It is a flowchart related to the first embodiment of the present invention;

[0060] Figure 3 It is a schematic diagram of the focus adjustment of the event camera related to the first embodiment of the present invention. The event camera focuses on a certain plane between the object to be measured and the camera to play a role in magnifying the speckle;

[0061] Figure 4 It is a schematic diagram of the laser speckle captured by the event camera related to the first embodiment of the present invention. The figure shows a good view of the change diagram of the speckle within two consecutive vibration periods of the object to be measured. The high similarity between the upper and lower two figures proves the principle of frequency consistency between the change frequency of the speckle pattern and the object vibration;

[0062] Figure 5This is the effect diagram after audio equalization filtering involved in the first embodiment of the present invention. The abscissa represents frequency, and the ordinate represents energy gain.

[0063] Figure 6 This is the spectrogram of the original audio and the vibration signal reconstructed by the method of the present invention in the first embodiment of the present invention. Among them, (a) is the spectrogram of the original audio; (b) is the spectrogram of the vibration signal reconstructed by the method of the present invention. Detailed implementation manners

[0064] The following combines the drawings to elaborate in detail on the specific implementation manners of the vibration perception method and system based on an event camera involved in the present invention.

[0065] <Example 1>

[0066] In this embodiment, the solution of the present invention is described by taking the restoration of a sound wave vibration signal as an example. As Figure 1 shown, the hardware platform built in this embodiment includes: an event camera, a sound source, a measured object, a laser emitter, and a computer terminal. The sound source includes a series of objects such as a speaker, a guitar, and a tuning fork that can generate sound waves and cause themselves or surrounding objects to vibrate. The measured object includes the sound source and a series of objects that vibrate themselves under the influence of the sound waves emitted by the sound source. The laser emitter is used to irradiate the vibrating measured object; the computer terminal is connected to the event camera; the event camera is used to collect event stream data and transmit it to the computer terminal. The event camera model is the DVXplorer event camera; the computer terminal is connected to the event camera through a USB3.0 to microUSB3.0 adapter.

[0067] As Figure 2 shown, the specific steps of the vibration perception method based on an event camera in this embodiment are as follows:

[0068] Step 1: In a static scene, use the laser emitter to irradiate a position with a low reflectivity of the measured vibrating object, and align the event camera with the laser spot on the measured object. The laser spot irradiated on the measured object by the laser emitter and the event camera should be on the same optical axis as much as possible.

[0069] Step 2: The event camera captures the moving image of the laser speckle. As Figure 3 shown, adjust the lens of the event camera so that it focuses on a certain plane between the measured object and the event camera, and capture the laser speckle event stream When a laser beam irradiates a rough surface, diffuse reflection occurs. Since the laser has strong coherence, different optical path differences are caused during the reflection process, resulting in interference and forming a randomly distributed granular speckle pattern. When the focal plane of the event camera is far from the object under test, the speckles can occupy a larger screen area, thus playing a magnifying role. Each unit of the event camera sensor works asynchronously and only outputs event points when the scene brightness changes. When there is a sound source in the scene, a slight deformation occurs on the surface of the object under test, causing a slight shift of the laser speckles on the focal plane and resulting in a brightness change, which can be detected by the event camera and output events. When there is no sound source in the scene, the speckle brightness remains unchanged and the event camera does not output events.

[0070] The imaging process of the event camera is as follows:

[0071]

[0072] In the formula, I(x,t) and I(x,t - τ) represent the brightness of pixel point x at times t and t - τ respectively. p j ∈{1, -1}, representing the polarity of the event point. δ(x,t j ) represents the event point triggered at x at time t j . c is a constant.

[0073] Let represent the event point triggered at x m at time t m , with its polarity being p m . is the set of all events, and M e is the total number of event points.

[0074] Step 3: Event stream denoising. The laser speckle pattern is concentrated in a certain area of the event camera's screen. Isolated event points are regarded as the inherent noise of the hardware and removed to obtain the denoised event stream. The specific method of event denoising is as follows:

[0075] Set the event number threshold λ thresh . For each incoming take its r×r neighborhood For all count the number of events falling within the neighborhood . If the number is less than λ, it is regarded as noise and removed. The denoised event stream is

[0076] Step 4: Time-domain sampling. The event stream is segmented according to the time stamp at the required audio sampling period to obtain the segmented event stream. The specific time-domain sampling method is as follows:

[0077] Set the segmented time window length Δt = 200 us. Different sampling rates can be obtained by adjusting the time window overlap ο of two consecutive events. For example, if the required sampling rate is 50000 Hz, the sampling period is 20 us. Then set ο = 180 us. At this time, it should be noted that the sampling frequency should satisfy the Nyquist sampling theorem. Obtain the segmented event stream where N ε is the total number of segments of the total event stream obtained by segmentation. Each segment of the event stream ε n contains bipolar events, that is, ε n ={ε n + , ε n -}, where:

[0078] ε n + ={e m |t m ∈[(n - 1)Δt, nΔt), p m = 1},

[0079] ε n - ={e m |t m ∈[(n - 1)Δt, nΔt), p m = -1},

[0080] During the experiment, taking Δt = 200 us and ο = 0 us has the best effect. At this time, the sampling frequency is 5000 Hz, which can meet the sampling requirements of most vibrations.

[0081] Step 5: Vibration signal reconstruction. Combining the principle of the frequency consistency of the speckle average brightness and the object vibration and the event camera imaging principle, count the number of positive event points n and the number of negative event points in each segment of the event stream ε Subtract the number of positive event points from the number of negative event points to obtain the reconstructed vibration signal The specific principle of audio reconstruction is as follows:

[0082] According to the attached figure Figure 3 shown, the two rows of this group of images are the speckle images of a single-frequency vibrating object in two consecutive vibration cycles. It can be seen that the corresponding upper and lower images are almost the same, proving that the average brightness of the speckle will change periodically depending on the vibration s(n) of the object to be measured, and the two should have the same vibration fundamental frequency.

[0083] In the nth segment of the event stream, for each pixel point x i of the event camera, there is:

[0084]

[0085] In an image, there are:

[0086]

[0087]

[0088] where N r is the resolution of the event camera. Let represent the geometric mean luminance of the image obtained at time t. On the right side of the equation, let:

[0089]

[0090]

[0091]

[0092] respectively represent the statistical values of the number of positive and negative event points in the nth segment of the event stream. It can be obtained that:

[0093]

[0094] That is, when the average luminance changes periodically, the left side of the equation will also change with the same period. Then also changes with the same period. Therefore, using the obtained signal contains the frequency information of the sound source audio vibration signal. The obtained reconstructed audio vibration signal:

[0095]

[0096] Step 6, vibration signal denoising. Set the noise signal threshold, and mark the signals in that are lower than this threshold as removed noise, and obtain the denoised vibration signal The specific method of vibration signal denoising is:

[0097] Set the noise signal threshold η thresh , and for the reconstructed audio signal set all values with amplitudes less than η thresh to zero, that is:

[0098]

[0099] In the experiment, set

[0100] Step 7, vibration signal equalization. According to the frequency distribution characteristics of the measured signal, design a filter bank e(n) to strengthen or weaken the energy of specific frequency bands. After equalization, the final vibration signal The specific method for vibration signal equalization is as follows:

[0101] Design an equalization filter bank e(n) to filter the denoised vibration signal That is:

[0102]

[0103] In the formula, * represents convolution.

[0104] Figure 5 shows the effect after vibration signal equalization filtering. It suppresses the low-frequency noise brought by alternating current below 80 Hz, suppresses the harmonic components above 600 Hz, and enhances the frequency band of 80 - 600 Hz where the musical sound is located.

[0105] According to Figure 6 the spectrogram of the original audio and the vibration signal reconstructed by the method of the present invention shown in, it can be seen that the two have the same time-frequency spectrum lines, which proves the reconstruction effect of the method of the present invention.

[0106] Visual vibration perception is a very useful method in the field of vibration perception, especially suitable for remotely capturing vibrations. Starting from the principle of an event camera, the present invention summarizes this solution through long-term experimental research. Use a laser to irradiate the object to be measured, and then use an event camera to capture the event stream of the reflected speckles. Based on the principle of the consistency between the average brightness of the speckles and the vibration frequency of the object and the imaging principle of the event camera, the vibration signal is reconstructed. The present invention explores the possibility in the field of vibration perception based on event cameras and summarizes a set of effective methods. And in the above embodiments, the feasibility of the algorithm is verified by taking the reduction of acoustic wave vibration signals as an example.

[0107] <Example Two>

[0108] Furthermore, in the second embodiment of the present invention, there is provided an event camera-based vibration perception system that can automatically implement the above method of the present invention. The system includes an operation prompt unit, an event stream acquisition unit, an event stream denoising unit, a time-domain sampling unit, a vibration signal reconstruction unit, a signal denoising unit, a signal equalization unit, an input display unit, and a control unit.

[0109] The operation prompt unit is used to prompt the operator: In a static scene, use a laser emitter to irradiate a position with a relatively low reflectivity of the vibrating object to be measured, and aim the event camera at the laser spot on the object to be measured; and prompt the operator to adjust the lens of the event camera so that it focuses on a certain plane between the object to be measured and the event camera for shooting.

[0110] The event stream acquisition unit uses an event camera to capture the moving image of the laser speckles and obtains the laser speckle event stream

[0111] The event stream denoising unit executes the content described in step 3 above. The laser speckle patterns are concentrated in a certain area of the event camera image. Isolated event points are regarded as inherent hardware noise and removed, obtaining a denoised event stream.

[0112] The time-domain sampling unit executes the content described in step 4 above. The event stream is segmented according to the time stamps at the required sampling period, obtaining a segmented event stream.

[0113] The vibration signal reconstruction unit executes the content described in step 5 above. It counts the number of positive event points n and the number of negative event points in each segment of the event stream ε. Subtract the number of positive event points from the number of negative event points. Obtain the reconstructed vibration signal.

[0114] The signal denoising unit executes the content described in step 6 above. Based on the noise signal threshold, signals below the threshold are marked as noise removed, obtaining a denoised vibration signal.

[0115] The signal equalization unit executes the content described in step 7 above. According to the frequency distribution characteristics of the measured signal, a filter bank e(n) is designed to strengthen or weaken the energy of specific frequency bands, and the final vibration signal is obtained after equalization.

[0116] The input display unit is used for the operator to input operation instructions and perform corresponding displays. For example, the input display unit can display the data, processing procedures, and output results obtained by each unit in the form of graphs (such as Figures 4 to 6 ) or tables for the operator to view.

[0117] The control unit is communicatively connected to the operation prompt unit, event stream acquisition unit, event stream denoising unit, time-domain sampling unit, vibration signal reconstruction unit, signal denoising unit, signal equalization unit, and input display unit to control their operations.

[0118] The above embodiments are merely illustrative examples of the technical solutions of the present invention. The vibration perception method and system based on an event camera involved in the present invention are not limited to the content described in the above embodiments, but are subject to the scope defined by the claims. Any modifications, supplements, or equivalent replacements made by those skilled in the art of the present invention based on this embodiment are within the scope protected by the claims of the present invention.

Claims

1. Event camera-based vibration sensing method, characterized in that, It includes the following steps: Step 1: In a static scenario, use a laser emitter to irradiate a position on the vibrating object to be measured with a relatively low reflectivity, and align the event camera with the laser spot on the object to be measured; Step 2: Use an event camera to capture the motion image of laser speckles; adjust the lens of the event camera so that it focuses on a certain plane between the object to be measured and the event camera, and capture the laser speckle event stream Step 3: The laser speckle pattern is concentrated in a certain area of the event camera's frame. Isolated event points are regarded as inherent hardware noise and removed to obtain a denoised event stream. Step 4, segment the event stream by timestamp according to the required sampling period to obtain the segmented event stream Step 5: Vibration signal reconstruction; Count the event stream ε for each segment n The number of positive event points And the number of negative event points Subtract the number of negative event points from the number of positive event points Obtain the reconstructed vibration signal Step 6, set the noise signal threshold, and mark the signals in that are lower than this threshold as noise removed, to obtain the denoised vibration signal Step 7. According to the frequency distribution characteristics of the signal under test, design a filter bank e(n) to enhance or weaken the energy of specific frequency bands, and obtain the final vibration signal after equalization Among them, in Step 5, the vibration reconstruction principle is: Within the n-th segment of the event stream, for each pixel x of the event camera i : where Δt represents the length of the segmented time window; p j ∈ {1, -1}, representing the polarity of the event point; δ(x i , t j ) represents the event point triggered at x i at time t j ; c is a constant; In an image: where N r is the resolution of the event camera; let denote the geometric mean luminance of the image obtained at time t, and on the right side of the equation, let: They respectively represent the statistical values of the number of positive and negative event points in the nth event stream. After simplification, it can be obtained: That is, when the average brightness changes periodically, the left side of the equation will also change with the same period, then also changes with the same period; therefore, using the obtained signal contains the frequency information of the sound source audio vibration signal, and the reconstructed audio vibration signal obtained:

2. The event camera-based vibration sensing method according to claim 1, characterized in that, It also includes: Among them, in Step 1, the laser spot irradiated by the laser emitter on the object to be measured and the event camera should be on the same optical axis as much as possible.

3. The event camera-based vibration sensing method according to claim 1, characterized in that: Wherein, In Step 4, the segmentation time window length Δt = 200 us and the time window overlap ο = 0 us are taken.

4. The event camera-based vibration sensing method according to claim 1, characterized in that: Wherein, In step 6, set the noise signal threshold Set all values with amplitudes less than η thresh to zero:

5. Event camera-based vibration sensing system, characterized in that, It includes: An operation prompt unit that prompts the operator to use a laser emitter to irradiate a position on the vibrating object to be measured with a relatively low reflectivity in a static scenario, and align the event camera with the laser spot on the object to be measured; and prompts the operator to adjust the event camera lens to focus on a certain plane between the object to be measured and the event camera for shooting; The event stream acquisition unit captures the laser speckle motion image using an event camera to obtain a laser speckle event stream Event stream denoising unit. The laser speckle patterns are concentrated in a certain area of the event camera screen. Isolated event points are regarded as the inherent noise of the hardware and removed to obtain the denoised event stream. The time-domain sampling unit divides the event stream according to the time stamp at the required sampling period to obtain the segmented event stream The vibration signal reconstruction unit counts each segment of the event stream ε n The number of positive event points And the number of negative event points Subtract the number of positive event points from the number of negative event points To obtain the reconstructed vibration signal A control unit that is communicatively connected to the operation prompt unit, the event stream acquisition unit, the event stream denoising unit, the time domain sampling unit, and the vibration signal reconstruction unit, and controls their operations; The signal denoising unit, which is communicatively connected to the control unit, based on the noise signal threshold, marks the signals in that are lower than the threshold as noise removed, and obtains the denoised vibration signal A signal equalization unit, communicatively connected to the control unit, designs a filter bank e(n) to strengthen or weaken the energy of a specific frequency band according to the frequency distribution characteristics of the measured signal, and obtains a final vibration signal after equalization Among them, in the vibration signal reconstruction unit, the vibration reconstruction principle is: Within the n-th segment of the event stream, for each pixel x of the event camera i : where Δt represents the length of the segmentation time window; p j ∈ {1, -1}, representing the polarity of the event point; δ(x i , t j ) represents the event point triggered at x i at time t j ; c is a constant; In an image: Where N r is the resolution of the event camera; let represent the geometric mean luminance of the image obtained at time t, and on the right side of the equation: They respectively represent the statistical values of the number of positive and negative event points in the nth event stream. After simplification, it can be obtained: That is, when the average brightness changes periodically, the left side of the equation will also change with the same period, then also changes with the same period; thus, by using the obtained signal contains the frequency information of the audio vibration signal of the sound source, and the reconstructed audio vibration signal is obtained:

6. The vibration perception system based on an event camera according to claim 5, wherein, It also includes: An input display unit that is communicatively connected to the control unit, and is used for the operator to input operation instructions and perform corresponding displays.

Images