A method for classifying targets through scattering media based on event cameras

Abstract

This invention discloses a target classification method based on an event camera using a scattering medium, belonging to the field of target detection technology. The invention captures the speckle pattern formed by a moving target in the optical path as it scatters through a scattering medium, preserving the target's features. When the target moves, its corresponding speckle pattern moves as well. An event camera is used to capture the event stream triggered by the moving speckle. The event stream is then denoised and normalized using the event's timestamp and polarity information to create event frames. These frames are then fed into a densely connected neural network to complete the classification task. Multiple comparative experiments have verified that "event speckle" retains target features for event streams of different lengths and moving targets in different directions, and the target classification method based on "event speckle" achieves a classification accuracy of up to 92.92%.

Description

Technical Field

[0001] This invention discloses a method for classifying moving targets through scattering media based on an event camera. It is suitable for classifying moving targets through scattering media in low-light environments. It belongs to the field of target detection technology. Background Technology

[0002] Unlike the propagation of light in free space, when light carrying target information passes through a scattering medium (such as clouds, smoke, or biological media), the inherent inhomogeneities of the medium cause scattering of the light, leading to wavefront distortion. At this point, the detector detects a complex scattering image, making it impossible for the human eye to directly perceive the target information. This phenomenon frequently occurs in daily life, scientific research, and security, affecting people's observation of objects. Therefore, how to obtain target information through scattering media has become an important and challenging problem.

[0003] After years of scientific research, imaging through scattering media has found wide applications in biomedical technology, astronomical observation, and everyday life. Researchers from different fields have proposed various methods, such as adaptive optics, wavefront shaping, correlation imaging, and the recently emerging neural networks. In all the aforementioned imaging applications, high-quality speckle images are required. However, in practical applications, it is unavoidable that the scattering medium or / and the target object are in motion. Summary of the Invention

[0004] The purpose of this invention is to solve the problem of classifying moving targets through scattering media under adverse conditions such as low brightness, and to provide a target classification method based on an event camera. This invention captures the speckle pattern formed by a moving target in the optical path through a scattering medium, preserving the target's characteristics. As the target moves, its corresponding speckle pattern moves as well. An event camera is used to capture the event stream triggered by the moving speckle. Compared to traditional cameras, event cameras have a higher dynamic range, can capture speckle patterns at lower brightness, and the events in the event stream have a microsecond-level temporal resolution, eliminating motion blur caused by target movement and excessively long exposure times in traditional cameras. The event stream is denoised, and normalized using event timestamps and polarity information to create event frames, or "event speckle," which are then fed into a densely connected neural network to complete the classification task. Multiple comparative experiments have verified that "event speckle" retains target characteristics for event streams of different lengths and moving targets in different directions, and the target classification method based on "event speckle" achieves a classification accuracy of up to 92.92%.

[0005] A target classification method based on an event camera through a scattering medium includes the following steps:

[0006] Step 1: Establish an event-based moving target speckle acquisition system. Illuminate the moving target with a light source. The beam of light reflecting from the moving target, carrying target information, is scattered by a scattering medium. An aperture is used behind the scattering medium to adjust the size of the transmitted speckle, which is then received by the event camera.

[0007] Step 2: Collect raw event speckle data generated by moving targets. Use an event camera to capture motion speckle data generated by moving targets; change different moving targets and their motion patterns, collect multiple sets of event data, and divide them into training and test sets.

[0008] Step 3: Denoise the coordinate and timestamp information of the multiple sets of event data collected in Step 2. Each set of event data includes the coordinates of a pixel, a timestamp, and the polarity of the event. Based on the timestamp, a segment of the event stream is extracted. For each triggering pixel within this segment, it is checked whether other events have triggered in adjacent positions on the same row. If so, the event is retained; otherwise, it is deleted. This process is repeated for all event streams to obtain the preprocessed event stream data.

[0009] Step 4: Normalize the preprocessed event stream data from Step 3 using the timestamps and polarity information of the event data to form event frames of moving targets passing through the scattering medium, making these event frames suitable for subsequent classification networks. Based on the timestamps, the event stream is segmented into equal time ranges; then, the segmented events are accumulated across a single frame to create an event frame. During the accumulation process, the event polarity p and timestamp t are used to normalize the events. The formula for normalizing the event timestamp is as follows:

[0010]

[0011] The event data collected from each moving target, after processing in step three, contains N events. Normalization aims to ensure that events triggered at different times have varying impacts within the event frame; the two polarities p represent two brightness variations. D represents the number of segments in the event data, and t... i Let t1 represent the timestamp of the i-th event in the event data, and t2 represent the timestamp of the 1-th event in the event data. N This represents the timestamp of the last event in the event data. This is a normalized timestamp.

[0012] The event data is divided into X segments of sub-data Sd, which are then used to create X event frames. The calculation formula for each event frame is as follows:

[0013]

[0014] Where i is the sequence number of each event; (x, y) are the coordinates of the event; p is the polarity of the event; S is the event subsequence; and d is the sequence number of the event subsequence (d = 0 when creating the first subsequence, d = 1 when creating the second subsequence, and d = 2 when creating the third subsequence).

[0015] The denoised event data is processed into the desired speckle event frames, or "event speckles," using the method described above, and then cropped to a size suitable for the classification network.

[0016] Step 5: Feed the event speckle data from Step 4 into a DenseNet-based event speckle classification network model for training. The classification network uses a densely connected network built with PyTorch, loading pre-trained weights from torchvision. Convolutional layers are added at the beginning to adapt the event speckle classification network model to different input data. At the end of the network, fully connected layers are used to transform the 1000 pre-trained classification results into the required number of categories, ultimately obtaining the classification results for the moving target.

[0017] Beneficial effects:

[0018] 1. This invention creatively proposes a method for classifying targets through a scattering medium using "event speckle". It employs an event camera to capture motion speckle patterns formed by a moving target passing through the scattering medium. The event data generated by the event camera is denoised and then processed into "event speckle" based on time and polarity information. Because events in the event stream have microsecond-level temporal resolution, motion blur caused by target movement and excessively long exposure times in traditional cameras can be eliminated. Therefore, the target classification method based on "event speckle" has a higher classification accuracy.

[0019] 2. This invention innovatively introduces event cameras into the field of scattering medium imaging, solving the problems of low brightness, high noise levels, and severe motion blur in traditional scattering medium imaging methods. Traditional cameras used for scattering medium imaging generally suffer from excessively low brightness and long exposure times. These long exposure times lead to severe noise and motion blur in the image, while also limiting the frame rate. Therefore, the recognition performance of traditional methods for identifying moving targets behind scattering media is severely limited. This invention utilizes the high dynamic range, low latency, and high temporal resolution of event cameras to build an optical system based on event cameras for target classification research through scattering media. It demonstrates good recognition results for both low-brightness speckle and speckle generated by moving targets. Furthermore, multiple experiments have verified that this invention achieves good recognition accuracy for moving event streams of different lengths and targets with different motion patterns, providing researchers in the field of scattering medium imaging with new ideas and methods for studying low-brightness moving targets. Attached Figure Description

[0020] Figure 1 This is a diagram of the data acquisition system for an event-based camera-based experimental setup.

[0021] Figure 2 This is the raw event stream that was collected;

[0022] Figure 3 The final "event speckle" created for the event stream. Detailed Implementation

[0023] To better illustrate the purpose and advantages of the present invention, the invention will be further described below in conjunction with the accompanying drawings and examples.

[0024] Example 1:

[0025] Unlike traditional cameras, event cameras feature high dynamic range, low latency, and high temporal resolution. Each pixel operates independently, detecting whether the brightness change at that pixel location exceeds a set threshold. If the threshold is exceeded, event information is output, including the pixel's position coordinates, the timestamp when the event was triggered, and polarity information. Event cameras boast a 120dB dynamic range and microsecond-level temporal resolution, allowing them to better adapt to low-brightness speckle effects caused by moving targets.

[0026] This invention constructs an optical path and uses an event camera to collect speckle event streams formed by moving targets passing through a scattering medium. After processing the event streams, it creates "event speckle" and feeds it into a neural network for training and testing. The resulting moving target classification results have a high accuracy rate.

[0027] To more clearly illustrate the above technical solutions, the technical solutions in the embodiments of the present invention will be described more completely below with reference to the accompanying drawings:

[0028] Step 1: See Figure 1 The experimental setup was located on an optical experimental platform. A 532nm Nd:YAG solid-state laser was used as the light source. The light source passed through a rotating frosted glass; the particle size of the frosted glass altered the spatial coherence of the laser, while the rotation altered its temporal coherence, together forming a narrowband spatially incoherent pseudothermal light source. The moving target was simulated using a digital micromirror array (DMD). An incoherent beam of light irradiated the DMD, which then loaded the moving target. The beam of light reflecting from the moving target, carrying target information, was scattered by a scattering plate (GCL-201102, 600 mesh). An aperture stop was placed close to the scattering medium to adjust the speckle size and the resolution of the imaging system. Finally, the scattered beam carrying target information was received by an event camera.

[0029] The specific method for loading moving targets using the DMD is as follows: Using the MNIST handwritten digit dataset, the original digit images are enlarged proportionally and placed on a pure white background to create binary targets (white background, black handwritten digits). These targets are then loaded using a digital micromirror array (DMD). Since the DMD works by rotating the array of micromirrors 12° left and right, when the DMD loads the binary image, the micromirrors corresponding to the black pixels flip towards the event camera, reflecting the light beam towards the camera. Conversely, the micromirrors corresponding to the white pixels flip to the opposite side. Thus, the event camera only receives the digit target information, and the relative pixel displacement of the digit targets between adjacent frames achieves the effect of target movement. The movement varies slightly depending on the size of the digits. In this example, the simulated motion is leftward, upward, and back to the original position, with each motion pattern loaded every 100ms. Multiple sets of event streams generated by the moving handwritten digit targets were collected to create "event speckle" patterns, which were then divided into training and test sets.

[0030] Step Two: The event camera uses a CeleX-V with a resolution of 1280×800. When capturing images using the event camera, if the brightness change exceeds a threshold, the event is read out and stored. The event output information includes the pixel's coordinate position, timestamp, and event polarity information, such as (x, y, p, t), representing the pixel's horizontal and vertical coordinates, polarity, and trigger time, respectively. If the increase in light intensity on a pixel exceeds a pre-set threshold (the threshold can be modified via the host computer interface), the polarity information p output for that event is +1; if the light intensity decreases and exceeds the threshold, the polarity information p output for that event is -1. The targets are repeatedly loaded until all the handwritten digit targets created on the digital micromirror array are loaded, and the collected event data is divided into training and testing sets.

[0031] Step 3: After obtaining the raw event stream data generated by the moving target through the scattering medium, the event stream is denoised using the coordinate information (x, y) and timestamp information t of the acquired event stream. For each group of moving targets, the event stream is divided into 300ms segments. For each triggered event within each 300ms segment, it is checked whether there are other triggered events in the adjacent positions of the same row. If so, the event is retained; otherwise, it is deleted. This step is repeated for all event streams to obtain the preprocessed event stream data.

[0032] Step 4: Obtain the preprocessed event streams, each containing three displacements of the moving target. To better represent "event speckle" and make it suitable for event speckle classification networks, we need to process the event streams into event frames.

[0033] The event sequence is divided into 100ms intervals. Then, all events within each segment are accumulated across a single frame to create an event frame. It's important to note that during the accumulation process, the event polarity (p) and timestamp (t) are used to perform a normalization-like process on the events.

[0034] The overall event stream is divided into three event streams, each containing a different motion pattern; each event stream includes a displacement. Then, the three event streams are accumulated on the screen, with each event stream creating a three-channel event frame, resulting in a total of nine-channel event frames. During the accumulation process, the event polarity p and timestamp t are used to perform a normalization-like process on the events. The normalized event timestamp formula is as follows:

[0035]

[0036] The goal is to ensure that events triggered at different times have different impacts within the event frame, where t i This represents the timestamp of the currently processed event, where t1 represents the timestamp of the first event in the event stream, and t... N This represents the timestamp of the last event in the event stream. D indicates how many segments the event stream is divided into; here, D = 3. Two polarities of p can represent two brightness variations. Each moving target corresponds to an event stream of N, which is then divided into three subsequences Sd, corresponding to three event frames. The calculation formula for each event frame is as follows:

[0037]

[0038] Where i is the sequence number of each event; (x, y) are the coordinates of the triggering event; p is the polarity of the event; S is the event subsequence; and d is the sequence number of the event subsequence (d=0 for the first subsequence, d=1 for the second subsequence, and d=2 for the third subsequence). The required event frames, i.e., "event speckle," are then created. The 1920×1200 event frames are then cropped to a size suitable for the classification network, such as 512×512.

[0039] Step 5: Feed the generated event speckle data into a DenseNet-based event speckle classification network model for training. The classification network uses a densely connected network built with PyTorch; in this example, DenseNet169 is used, loaded with pre-trained weights trained by torchvision. A convolutional layer with a kernel size of 1 and a stride of 1 is added at the beginning, without changing the kernel size, only changing the number of image channels (e.g., H×W×3C is modified to H×W×C after convolution). GELU is used as the activation function. The number of convolutional layers can be adjusted appropriately to adapt the network to event speckle inputs with different channels. The fully connected layer of DenseNet169 outputs preliminary classification results for 1000 classes. Subsequent fully connected layers convert the 1000 classes into 10 classes. The resulting classification results are shown in Table 1.

[0040] Target movement direction Event stream capture duration Classification accuracy 1 Left 100ms 75.00％ 2 Above 100ms 82.08％ 3 bottom right 100ms 81.25％ 4 Left + Top + Bottom Right 100ms 89.58％

[0041] Example 2

[0042] Comparative Example 1: Steps two through three and step five are the same, except that in step one, the DMD loads the moving target at half the original speed, meaning one movement of the target is completed in 200ms. In step four, the event data is divided into segments with an event length of 200ms, meaning each subsequence contains more events. A comprehensive comparison of the results obtained in Example 2 and Example 1 is shown in Table 2:

[0043] Target movement direction Event stream capture duration Classification accuracy 1 Left 100ms 75.00％ 2 Left 200ms 76.67％ 3 Above 100ms 82.08％ 4 Above 200ms 90.83％ 5 bottom right 100ms 81.25％ 6 bottom right 200ms 92.50％ 7 Left + Top + Bottom Right 100ms 89.58％ 8 Left + Top + Bottom Right 200ms 92.92％

[0044] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for classifying targets through scattering media based on an event camera, characterized in that: Includes the following steps: Step 1: Build an event-based moving target speckle acquisition system; A moving target is illuminated by a light source. The light beam reflecting the target information is scattered by a scattering medium. An aperture is used to adjust the size of the transmitted speckle after the scattering medium, and then the light is received by the event camera. Step 2: Collect raw event speckle data generated by moving targets; use an event camera to capture motion speckle generated by moving targets; change different moving targets and their motion patterns, collect multiple sets of event data, and divide them into training and testing sets; Step 3: Denoise the coordinate position information and timestamp information of the multiple sets of event data collected in Step 2: Each set of event data includes the coordinate position of the pixel, the timestamp, and the polarity of the event; Extract an event stream within a certain period based on the timestamp. For each triggered pixel in the event stream, check if there are other events triggered in the adjacent positions of the same row as the pixel. If there are, keep the event; otherwise, delete the event. Repeat this check for all event streams to obtain the preprocessed event stream data. Step 4: Normalize the event stream data after the preprocessing in Step 3 using the timestamp and polarity information of the event data to form event frames of moving targets passing through the scattering medium, so that the event frames are suitable for subsequent classification networks; Based on timestamps, the event stream is divided into segments with the same time range; The segmented events are then accumulated on a single frame to create an event frame. During the accumulation process, the event polarity p and timestamp t are used to normalize the event. The formula for normalizing the event timestamp is as follows: The number of events contained in the event data collected from each moving target after processing in step three is N; the purpose of normalization is to ensure that events triggered at different times have different influences in the event frame, and the two polarities p represent two brightness changes; D represents the number of segments of event data, t i Let t1 represent the timestamp of the i-th event in the event data, and t2 represent the timestamp of the 1-th event in the event data. N This represents the timestamp of the last event in the event data. This is a normalized timestamp; The event data is divided into X segments of sub-data Sd, which are then used to create X event frames. The calculation formula for each event frame is as follows: Where i is the sequence number of each event; (x, y) are the coordinates of the event; p is the polarity of the event; S is the event subsequence; d is the sequence number of the event subsequence, where d = 0 when the first subsequence is created, d = 1 when the second subsequence is created, and d = 2 when the third subsequence is created; The denoised event data is used to create the desired speckle event frames, i.e., "event speckle", from the above method, and then cropped to a size suitable for the classification network. Step 5: Feed the event speckle data from Step 4 into the DenseNet-based event speckle classification network model for classification training; The classification network uses a densely connected network built on PyTorch, loaded with pre-trained weights trained by torchvision. By adding convolutional layers at the beginning, the event speckle classification network model is adapted to different input data. At the end of the network, a fully connected layer is used to convert the 1000 pre-trained classification results into the required number of categories, and finally, the classification results for moving targets are obtained.

Images