Microscopic light splitting type event camera and traditional camera alignment system

Abstract

The invention provides a microscopic light splitting type event camera and traditional camera alignment system. According to the system, a microscopic light path module guides the same light beam to an event camera and a traditional camera at the same time according to a set splitting ratio; the hardware trigger control module respectively accesses external trigger pulses to the event camera and the traditional camera; the time synchronization management module performs link delay compensation on the event camera and the traditional camera; the data acquisition module is used for acquiring an event graph of the event camera and an image frame of the traditional camera in parallel; the space alignment module processes the image frame to enable the image frame to be aligned with an event graph of an event camera, and the aligned image frame and event graph are obtained; and the fusion detection module is used for sending the aligned image frame and event graph into a fusion detection network for fusion detection to obtain a detection result. According to the system, modal mismatching and training noise caused by soft synchronization and rough alignment are avoided, so that fusion detection has stable, reviewable and engineered deployment capabilities.

Description

Technical Field

[0001] This invention relates to the fields of machine vision, microscopic imaging and intelligent detection technology, specifically to a microscopic spectroscopic event camera and a traditional camera alignment system. Background Technology

[0002] Microscopic inspection technology plays a fundamental supporting role in precision manufacturing, electronic semiconductors, biomedicine, and materials characterization. Its development has undergone a continuous evolution from traditional optical microscopy to digital imaging and intelligent inspection. The introduction of high-resolution industrial cameras, programmable illumination and motion platforms, as well as image processing and deep learning algorithms, has significantly improved the spatial resolution, automation level, and inspection throughput of the system. However, in microscopic scenarios characterized by high contrast, high reflectivity, rapid transients, and microscale, the dynamic range and temporal sampling capability of the imaging link still constitute bottlenecks. Especially in online production and high-volume inspection tasks, the requirements for real-time performance, stability, and verifiability are becoming increasingly stringent, which challenges the applicability of traditional single imaging mechanisms in complex working conditions.

[0003] In the existing technological system, traditional frame-based cameras, based on a synchronous imaging mechanism of "exposure-readout-frame output," can provide absolute brightness and high-fidelity texture. However, their dynamic range and temporal resolution are inherently constrained by their structure: the effective dynamic range of conventional industrial cameras is typically around 60–70 dB. When facing highly reflective samples such as metal mirrors, polished aluminum, or coated optical components, bright areas are prone to saturation and detail compression. If exposure is shortened to suppress highlights, dark areas are prone to underexposure, making it difficult to capture all the details within a single frame. In the temporal dimension, typical frame rates are around 30–120 fps, corresponding to exposure and readout timescales in the millisecond range (approximately 1–10 ms). Fast motion or transient phenomena can easily cause motion blur or undersampling. If a rolling shutter is used, interline time difference and geometric distortion may also occur, further affecting high-precision measurement and positioning. Traditional remedies for highly reflective and high-contrast scenes (such as polarization suppression and multi-exposure HDR) have limitations in applicability and real-time performance under dynamic tasks, making it difficult to fundamentally eliminate detail loss and artifacts caused by saturation.

[0004] In contrast, the Dynamic Vision Sensor (DVS) employs an asynchronous, pixel-by-pixel triggering mechanism driven by logarithmic brightness and a threshold: each pixel independently compares the current logarithmic brightness with the logarithmic brightness difference at the time of the previous event; when the difference exceeds the threshold, an event with polarity and a timestamp is generated. Unaffected by global exposure and progressive readout limitations, this mechanism enables the DVS to achieve microsecond-level temporal resolution and low latency, and it can stably respond to local brightness changes even under high dynamic range conditions (often exceeding 120dB), making it suitable for capturing high-speed edges and transient details. In recent years, DVS has been explored for applications in machine vision, including high-speed counting, target tracking, rapid defect detection, and HDR scene analysis; however, its output represents a sparse event stream of brightness changes, which is insufficient for directly expressing static textures and low-frequency, slowly varying information. Using it alone is insufficient to meet the demands for high-resolution morphology and quantitative measurement in microscopic inspection.

[0005] To balance dynamic and static information, existing solutions attempt to fuse frame-based and event-based cameras. These approaches include temporal alignment based on software timestamps followed by late-stage feature fusion, spatial alignment based on image registration followed by early-stage channel fusion, or fixed cropping / rotation / scaling after a single calibration. However, these approaches generally suffer from insufficient spatiotemporal consistency: software synchronization relies on system clocks and thread scheduling, with jitter typically on the order of milliseconds, making it difficult to meet the microsecond-level alignment requirements of dynamic microscopic processes; spatial alignment, if relying solely on single calibration or global homography, is easily affected by sensor resolution, pixel size, and assembly micro-errors in spectroscopic microscopy, leading to persistent residuals in field of view ratio, angle, and center position; simultaneously, highly reflective or weakly textured areas cause unstable feature point registration, making geometric misalignment unavoidable. Inaccurate alignment introduces modal mismatch noise during training and inference, affecting recall, localization accuracy, and model convergence stability, thus limiting the usability of the fusion solution in online and high-volume detection.

[0006] From a technical perspective of time synchronization and spatial alignment, existing time alignment can be divided into software synchronization and hardware synchronization. The former uses the system timestamp as a benchmark, which is easily affected by factors such as operating system scheduling, buffering, and load fluctuations, making it difficult to achieve microsecond-level and long-term stable consistency. The latter unifies the acquisition starting point through external triggering, and can achieve microsecond-level consistency under the condition that the trigger source is stable, but requires additional trigger sources, connections, and camera-end trigger configuration. Common spatial alignment methods include geometric mapping based on calibration plates and homography, image registration based on feature points or contours, and cropping and rotation scaling based on geometric priors. Due to the differences in resolution and pixel size between the two sensors after beam splitting, the lens-camera assembly may also produce slight angles and center offsets. In addition, the events and the traditional two types of image representations are significantly different, and general methods are prone to failure in highly reflective or weakly textured areas; one-time calibration alone is also difficult to cover batch differences and long-term drift. Therefore, in the engineering context of highly reflective materials and microscopic dynamic tasks, the industry's demand for "microsecond-level time synchronization + pixel-level spatial alignment + fusion detection robust to reflection and low texture" continues to increase. There is an urgent need to establish a full-link engineering solution from optical path to time, from space to data and model, so as to provide reliable support for high-precision and high-stability detection under complex working conditions. Summary of the Invention

[0007] To address the aforementioned problems in the prior art, this invention provides a microscopic beam-splitting event camera and a conventional camera alignment system. The technical problem to be solved by this invention is achieved through the following technical solution: A microscopic spectroscopic event camera and conventional camera alignment system includes: The microscopic optical path module is used to simultaneously guide the same light beam to both the event camera and the conventional camera according to a set beam splitting ratio. The hardware trigger control module is used to generate external trigger pulses and connect them to the event camera and the traditional camera respectively, so that the acquisition start point of both is locked at the same pulse trigger time. The time synchronization management module is used to perform link delay compensation on event cameras and traditional cameras respectively, using a unified timestamp as the time reference, so that the data generated by both at the same time has the same timestamp. The data acquisition module is used to acquire the event map of the event camera and the image frame of the traditional camera in parallel under the unified external trigger pulse and the time reference, and store the image frame and the event map into the corresponding cache queue; The spatial alignment module is used to align the image frames of the traditional camera with the event map of the event camera by using the relevant parameters of the event camera as a target reference and processing them through affine transformation and intelligent cropping, so as to obtain the aligned image frames and event map. The fusion detection module is used to feed the aligned image frames and event graphs into the fusion detection network for fusion detection to obtain the detection results.

[0008] Beneficial effects: This invention offers significant advantages on three levels: First, hardware-level synchronization ensures consistent acquisition starting points at the physical layer, significantly reducing jitter introduced by software clocks and scheduling, achieving microsecond-level time consistency; second, two-stage spatial alignment combines analytical geometry with robust statistics, enabling stable solutions for rotation, scale, and center parameters even under conditions of noise, reflection, and slight underexposure, achieving pixel-level geometric consistency; third, strictly aligned dual-modal fusion significantly improves recall, positioning accuracy, and robustness for highly reflective materials and dynamic microscopic scenes, while the model results possess good interpretability and engineering maintainability.

[0009] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of a microscopic spectroscopic event camera and a conventional camera alignment system provided by the present invention. Detailed Implementation

[0011] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0012] This invention aims to address the problems of traditional frame cameras in microscopic inspection scenarios, such as overexposure under strong reflection and rapid transient conditions, insufficient detail compression and time sampling, and the inadequacy of event cameras in expressing static textures and low-frequency slowly varying information. It provides a dual-modal alignment and fusion detection scheme that achieves microsecond-level time synchronization and pixel-level spatial consistency under the same optical path, avoiding modal mismatch and training noise caused by soft synchronization and coarse alignment, and enabling fusion detection to have stable, verifiable and engineerable deployment capabilities.

[0013] This invention provides a microscopic beam splitting event camera and traditional camera alignment system. The overall system architecture uses a single microscope head and beam splitter, allowing the same light beam to be projected onto the photosensitive surfaces of the event camera and the traditional frame camera according to a set splitting ratio, forming optical path and scene consistency. The host computer software is equipped with a hardware synchronization control module and a unified timestamp management module to manage the timing of the two cameras in parallel. The spatial alignment module uses the event camera as a reference to align the traditional camera image to the same resolution and field of view after affine transformation and intelligent cropping. The fusion detection module inputs the aligned dual-modal data into a target detection network (such as the YOLO series) for training and inference according to a predetermined fusion strategy.

[0014] like Figure 1As shown, the present invention provides a microscopic spectroscopic event camera and a conventional camera alignment system comprising: The microscopic optical path module 100 is used to simultaneously guide the same light beam to an event camera and a conventional camera according to a set beam splitting ratio; wherein, the microscopic optical path module includes a microscope lens and a beam splitter, the microscope lens and the beam splitter working together to guide the same light beam to the event camera and the conventional camera simultaneously according to a set beam splitting ratio.

[0015] The conventional camera of this invention is an RGB camera, and the event camera is a three-dimensional camera, with the three dimensions being the X-axis, Y-axis, and time axis.

[0016] The event camera is an asynchronous visual sensor triggered by changes in brightness. Each pixel independently compares the current logarithmic brightness with the logarithmic brightness at the time of the previous event. When the difference exceeds a threshold, an event with polarity and a timestamp is generated at the pixel coordinates, and the output is a sparse temporal event stream. Because it is not limited by global exposure and progressive readout, the event camera has microsecond-level temporal resolution and low latency. Furthermore, its logarithmic brightness perception and threshold triggering mechanism give it a significantly higher dynamic range than traditional cameras (often exceeding 120dB), enabling it to stably output edge and change information even in scenes with strong reflections, high contrast, and fast movement.

[0017] Traditional cameras output timed frames using a synchronous "exposure-readout-frame synthesis" mechanism, providing absolute brightness and high-fidelity textures, but with a trade-off between frame rate and exposure time, limiting dynamic range. Event cameras output event streams using a "threshold-asynchronous-pixel-by-pixel" mechanism, providing temporal details of changes and high dynamic response, but with limitations in expressing static textures and low-frequency gradual changes. By fusing these two approaches based on beam splitting and precise alignment, static morphology and dynamic changes can be simultaneously acquired within the same pixel grid and temporal sequence, achieving complementary advantages.

[0018] The hardware trigger control module 200 is used to generate external trigger pulses and connect them to the event camera and the traditional camera respectively, so that the acquisition start point of both is locked at the same pulse trigger time. The time domain uses a unified external trigger as the physical reference: the hardware synchronization control module communicates with the development board through a serial port and sends start / stop commands and trigger frequency parameters according to the configuration; the development board outputs a stable pulse (e.g., 20Hz, which can be configured as 1–100Hz), which is connected to the event camera trigger input and the traditional camera external trigger line (e.g., Line0) respectively to unify the acquisition starting point.

[0019] The time synchronization management module 300 is used to perform link delay compensation on event cameras and traditional cameras respectively, using a unified timestamp as the time reference, so that the data generated by both cameras at the same time has the same timestamp. The time synchronization management module records a base timestamp at the arrival time of each trigger and sets adjustable offsets for DVS and DV to compensate for link latency differences. It also tracks trigger counts, average intervals, and the most recent trigger time for stability monitoring. This solution eliminates jitter caused by software clocks and thread scheduling at the physical level, achieving microsecond-level cross-modal time consistency. Furthermore, it automatically degrades to software trigger mode to generate external trigger pulses in case of serial port anomalies or device disconnection (when the hardware trigger control module is inoperable), ensuring continuous process operation.

[0020] The data acquisition module 400 is used to acquire the event map of the event camera and the image frame of the traditional camera in parallel under the unified external trigger pulse and the time reference, and store the image frame and the event map into the corresponding cache queue; The spatial alignment module 500 is used to align the image frames of the traditional camera with the event map of the event camera by using the relevant parameters of the event camera as a target reference and processing them through affine transformation and intelligent cropping to obtain aligned image frames and event maps. The fusion detection module 600 is used to send the aligned image frames and event graphs into the fusion detection network for fusion detection to obtain the detection results.

[0021] The fusion detection network uses the YOLO series network, including Backbone, FPN / PAFPN, and Head module.

[0022] This system forms a closed-loop structure from top to bottom: "optical consistency—temporal synchronization—spatial alignment—fusion detection." A single microscope head and beam splitter ensure that both sensors observe the same field of view, achieving a balance between luminous flux and signal-to-noise ratio under engineering constraints of the splitting ratio. Unified hardware triggering binds the acquisitions of the two cameras to the same pulse source. Timestamp management uses the trigger time as a unified benchmark, setting adjustable offsets for different link delays and providing stability indicators such as trigger count, average interval, and most recent trigger time. Spatial alignment uses the field of view of the event camera as the target reference, performing affine corrections (rotation, scaling, and translation) on the traditional camera image and cropping it to the same resolution, ensuring that the two cameras are co-located at the pixel level and maintain a geometric isomorphic relationship. Fusion detection performs cross-modal feature learning under the aforementioned consistency conditions. It preferably adopts channel-dimensional cascade fusion at the input level, merging the aligned traditional image and event representation in the channel dimension to form a unified multi-channel input tensor, which is then input into the YOLO series network for training and inference. The system can also perform feature-level fusion at the network neck and detection head. Both maintain geometric-temporal alignment without being destroyed under a unified labeled coordinate system and synchronized data augmentation strategy. This organically integrates theoretical principles such as similarity transformation, robust statistics, and temporal synchronization with engineering aspects such as serial communication, parameter recording, batch processing, and visualization, and combines interpretability, verifiability, and maintainability.

[0023] In one specific embodiment of the present invention, the hardware trigger control module includes a serial communication unit, a trigger control unit, and a trigger parameter configuration unit; The serial communication unit is used to communicate with external modules; the trigger parameter configuration unit is used to set the trigger frequency, start command, and stop command; the trigger control unit is used to generate external trigger pulses according to the trigger frequency under the start command, and to stop generating the external trigger pulses under the stop command. The trigger frequency is configurable within the range of 1 to 100 Hz.

[0024] In one specific embodiment of the present invention, the time synchronization management module includes a hardware-triggered timestamp recording unit, an offset calibration unit, and a statistical analysis unit; The hardware trigger timestamp recording unit is used to record the trigger time of the external trigger pulse; The offset calibration unit is used to set adjustable offsets for compensating for link delay for the event camera and the traditional camera respectively, and to calibrate the event camera and the traditional camera according to the adjustable offsets. The statistical analysis module is used to count the trigger count, the average interval between trigger times, and the most recent trigger time, and to continuously monitor these three factors. When the hardware trigger control module fails to function, an external trigger pulse is generated using external software.

[0025] In one specific embodiment of the present invention, the data acquisition module includes a data acquisition subunit, an event representation construction subunit, and a preprocessing subunit; The data acquisition subunit is used to acquire the event stream of the event camera and the initial image frames of the traditional camera in parallel under a unified external trigger pulse and the time reference. The traditional image processing chain acquires scintillation rectangular pattern sequences from conventional cameras under a microscope. The conventional image processing chain uses LAB spatial L-channel enhancement and adaptive histogram equalization, followed by Otsu thresholding and morphological opening and closing to stabilize the contours. The event image processing chain uses grayscale conversion, Gaussian smoothing and fixed threshold binarization, combined with morphological opening and closing to suppress noise.

[0026] The event representation construction sub-unit is used to assemble the event flow into an initial event graph; The preprocessing subunit is used to perform channel enhancement and adaptive histogram processing on the initial image frame, then perform Otsu thresholding and morphological opening and closing to stabilize the contour to obtain the image frame, and sequentially perform grayscale, Gaussian smoothing and fixed threshold binarization processing on the initial event map, and cooperate with morphological opening and closing to obtain the event map; and store the image frame and event map into the corresponding cache queue.

[0027] In one specific embodiment of the present invention, the spatial alignment module includes: a coarse alignment submodule and a fine alignment submodule; The coarse alignment submodule is used to estimate the initial scaling range of the conventional camera relative to the event camera, using the field of view, pixel size, and resolution of the event camera as a target reference. The precision alignment module is used to calculate the angle difference, uniform scale, and center translation relative to the target reference and within the initial estimation range using a rotating rectangle detection and median robust estimation method; and to perform an affine transformation on the image frame according to the angle difference, uniform scale, and center translation, and to perform intelligent cropping and mirror boundary padding to obtain the aligned image frame and event map.

[0028] In a specific embodiment of the present invention, the precision alignment module is used to extract contours from the event graph and the aligned image frame and fit them with a minimum bounding rectangle, filtering out contours with abnormal aspect ratios to obtain the rotation rectangles corresponding to the event graph and the image frame respectively; calculate multiple angle differences between the rotation matrices corresponding to the event graph and the image frame respectively, and select the median of the multiple angle differences; select the event graph and image frame closest to the median as the representative frame pair; calculate the uniform scale factor s and the translation vector Δc based on the representative frame pair: let the width and height of the rotation rectangle representing the event graph be (w_e, h_e), and the width and height of the rotation rectangle representing the traditional image be (w_t, h_t), then the uniform scale factor is defined as s = Let the coordinates of the center of the event graph's representative contour be c_e, and the coordinates of the center of the traditional image's representative contour be c_t. Then, the translation vector Δc = c_e - c_t. Based on the angle difference of the representative frame pair, the calculated uniform scale s, and the translation vector Δc, all image frames are rotated, scaled, and translated accordingly. Then, all translated image frames are intelligently cropped and mirrored to obtain aligned image frames and event graphs.

[0029] The spatial domain uses the field of view of the event camera as the target reference and employs a two-stage strategy of "coarse alignment + fine alignment". The coarse alignment derives the field-of-view ratio based on the resolution, pixel size, and magnification of the two cameras, specifically through a formula. The initial scaling factor of the conventional camera relative to the event camera (P_t and P_e are the pixel sizes of the conventional camera and the event camera, respectively) is calculated, and the initial scaling range [s0(1-δ), s0(1+δ)] is given in combination with mechanical assembly tolerances, where δ is the tolerance coefficient, taken as 0.05 and a small initial angle range, as the initial convergence value for fine alignment; fine alignment involves acquiring a scintillation rectangular pattern sequence under a microscope, and enhancing and thresholding the two images respectively (the conventional image can be taken from the L channel of the LAB space and use contrast-limited adaptive histogram equalization and Otsu thresholding; the event image is grayscaled, lightly Gaussian smoothed and fixed thresholded), combined with morphological opening and closing to suppress noise; the binary image is contour extracted and fitted with the minimum bounding rectangle to obtain the rotation angle, center and axis aligned bounding box of each frame, the angle is uniformly standardized to [-90°, 0°] and stabilized according to the width and height exchange rule; the median of the angles of multiple frames is taken as the representative angle, and the frame closest to the median is selected as the representative rectangle. Using the event camera as a reference, the angular difference Δθ between the traditional camera and the event camera is calculated. First, the traditional camera image is rotated. Then, a uniform scale factor s and a translation vector Δc are calculated based on a representative rectangle as described above. Next, the rotated image is scaled proportionally based on s, and a translation based on Δc is performed. This yields the affine transformation T = Crop∘Trans(Δc)∘Scale(s)∘Rot(Δθ), where Crop performs intelligent cropping based on the event camera resolution and axis-aligned bounding boxes, and uses mirrored boundary padding to reduce extrapolation artifacts. This method combines analytic geometry with robust statistics, ensuring that rotation, scaling, and centering errors remain stable and solvable under noise and reflection conditions, and achieving pixel-level co-addressability. To meet engineering traceability requirements, Δθ, s, Δc, and representative rectangle parameters for each batch are recorded to a CSV file, and an inverse transformation is provided to map the aligned coordinates back to the original high-resolution traditional image coordinate system.

[0030] In one specific embodiment of the present invention, the fusion detection module includes a dual-modal fusion module and a defect detection module; The dual-modal fusion module is used to fuse the event graph with the aligned image frame to obtain a fusion tensor for detection; The defect detection module is used to input the fused tensor into the detection head to obtain the defect detection result. The defect detection result includes: a defect box marking the defect location, the defect category, and the confidence level of the corresponding defect category.

[0031] Traditional images can be grayscale or RGB. Fusion strategies include: early fusion, which concatenates the two modalities into a multi-channel input along the channel dimension and feeds it directly into the YOLO backbone network; and mid-to-late-stage fusion, which uses two branches to extract features separately and performs feature-level fusion at the network neck (e.g., FPN / PAFPN) or the detection head, and can introduce channel attention or learnable weighting to improve the effective channel contribution. During the training and inference phases, data augmentation strategies consistent with geometric-temporal alignment should be maintained: geometric augmentation (scaling, flipping, affine) should be applied simultaneously on both modalities, color augmentation should be limited to the traditional image branch to maintain the event temporal semantics; annotations should be shared in the aligned unified coordinate system, and the loss can be a combination of CIoU / DFL with classification and confidence losses, with appropriate weighting according to modal contribution to obtain a stable learning signal.

[0032] In the inspection of highly reflective materials, traditional images are prone to brightness compression and saturation in bright areas. However, event cameras, with their high dynamic range and sensitive response to brightness changes, can still output reflective edges and subtle dynamic clues. The fused network simultaneously utilizes high-fidelity textures in unsaturated areas and structural and dynamic information from the event channel to reduce false negatives and missed detections caused by overexposure. In microscopic inspection scenarios, the microsecond-level temporal resolution of event cameras can capture transient processes (such as microcrack initiation, particle motion, and droplet boundary oscillation), while traditional cameras provide static morphology and dimensional measurements. With strict geometric and temporal consistency guarantees, dual-modal input allows the network to learn dynamic and static information simultaneously without sacrificing spatial resolution, making it suitable for tasks such as online process monitoring, defect classification, and high-precision measurement.

[0033] In terms of technical principles and working mechanisms, the beam splitter projects the same beam onto two sensor planes simultaneously at a set ratio. Under conditions of small viewing angle and coaxiality, the two imaging paths approximately share a similar projection model. The resolution and pixel size of the sensors determine the spatial sampling frequency, thus affecting the effective field of view and texture resolution. Traditional cameras obtain frame images through an exposure-readout mechanism, expressing absolute brightness and detailed texture; event cameras trigger events through brightness change thresholds, outputting the temporal sequence and edge structure of brightness changes, possessing high dynamic range and high temporal resolution. Under the same optical path and time synchronization conditions, both have spatial projection consistency and temporal correspondence in their response to the same physical event, forming complementarity in fusion. Temporally, external hardware triggering serves as a unified physical reference, locking the acquisition start point of both types of cameras at the same pulse edge moment; after triggering, the event camera continuously generates an event stream based on brightness changes, while the traditional camera starts the exposure cycle when the trigger edge arrives. Due to the influence of electronic links and driving mechanisms, the data timestamps of the two channels may have a fixed offset. Unified timestamp management uses the trigger time as a benchmark, sets adjustable offsets for different channels to compensate for differences, and evaluates stability using indicators such as trigger count, average interval, and most recent trigger time. Compared with software timestamps, hardware triggering significantly reduces system scheduling jitter, achieving time consistency at the microsecond level. Spatially, under small-viewpoint approximation, the mapping from a traditional camera to an event camera can be described by a two-dimensional affine transformation, consisting of rotation, unified scaling, and translation. This invention utilizes the geometric structure of scintillation rectangular patterns under a microscope to obtain robust parameters: the rectangular boundaries have clear linear features and high edge contrast; the minimum bounding rectangle fits stably to output rotation angles and axis-aligned bounding boxes; the median of the temporal angle sequence is used to suppress anomalies; the scale factor is taken as the smaller value of the aspect ratio of the two axis-aligned bounding boxes to avoid content clipping; the translation amount is obtained from the center difference of the rotated and scaled rectangles, and intelligent clipping is performed at the event camera resolution to cover the effective field of view.

[0034] In terms of the engineering implementation and maintenance strategy for time alignment, the system uses an external pulse as a unified reference. It communicates with the development board via serial port to configure the port and baud rate and issue start / stop commands. The development board outputs pulses at a preset frequency (e.g., 20Hz, configurable from 1–100Hz), with the rising or falling edge connected to the event camera trigger input and the traditional camera's external trigger circuit, respectively. The event camera opens its trigger input via the SDK, while the traditional camera sets its trigger mode, trigger source, and trigger edge polarity, with an optional overlap mode to adapt to exposure readout. The initialization phase completes the camera trigger parameter settings and serial port establishment; the startup phase enables triggering; the running phase records statistical information and monitors for anomalies; and the stopping phase issues a stop command and releases resources. Unified timestamp management records the reference at each trigger moment, setting adjustable offsets for "DVS" and "DV" to compensate for link differences, while providing trigger counts, average intervals, and the most recent trigger moment for stability evaluation. If the serial port fails or the device disconnects, the system will issue an error signal and automatically degrade to software timestamp mode to ensure process continuity. For maintenance, it is recommended to regularly check the mean and variance of the trigger interval, verify the stability of the offset parameters, check the connection and port permissions, and recalibrate the offset with a standard test sequence when necessary to ensure long-term reliability of the synchronization link.

[0035] In terms of the engineering implementation, parameter setting, and inverse mapping of spatial alignment, a two-stage strategy of coarse-to-fine alignment is adopted. Coarse alignment calculates the field-of-view ratio based on the resolution, pixel size, and magnification of the two sensors to obtain the initial scaling range of the traditional camera relative to the event camera. It also incorporates mechanical assembly tolerances to provide a small initial angle range to narrow the search space for fine alignment. In the fine alignment stage, contours of the image frames and image frames generated by the event stream are extracted and fitted with the minimum bounding rectangle. Contours with excessively small areas or abnormal aspect ratios are filtered out to obtain their respective rotation rectangle parameters. Multiple angle differences between the rotation rectangles are calculated, and the median of these angle differences is used to mitigate anomalies. The two frames closest to the median are selected as representatives. Using the event camera as a reference, the angle difference between the traditional camera and the event camera is rotated. Based on the two representatives, a unified scale is calculated according to the aspect ratio for scaling. Then, a translation vector is calculated based on the center difference. Finally, intelligent cropping is performed within the event camera resolution and axis-aligned bounding box, using mirrored boundary filling to reduce interpolation artifacts. In the batch processing stage, alignment parameters for each frame are recorded to a CSV file, and an alignment visualization overlay is output for manual verification. For individual abnormal frames, fine-tuning can be performed based on representative parameters. To support the backfilling of the original high-resolution image, an inverse transformation calculation method is provided to map the alignment coordinates back to the original traditional image coordinate system, ensuring consistency between downstream measurement and process positioning. Regarding parameter recommendations, traditional images can use a moderate CLAHE (e.g., clipLimit≈3.0, tileGrid≈8×8), while event images can use a fixed threshold (e.g., 200) and perform opening and closing operations using 5×5 elliptical structural elements. The rotation rectangle angle is standardized to [-90°, 0°] and stabilized using a width-height exchange strategy. The median value is used as the representative angle value, and the scale factor is the smaller of the width-height ratio to avoid content cropping. The translation amount is obtained based on the center difference after rotation and scaling.

[0036] In terms of the implementation and training inference of bimodal fusion and YOLO defect detection, fusion detection is carried out after alignment, under the premise of geometric and temporal consistency. Event data representation can be accumulated into event frames (with polarity channels) within the trigger window, construct a time surface by exponential decay of event time, or generate multi-channel event maps according to multiple time windows; traditional images can be grayscale or RGB. Early fusion adopts channel-dimensional cascade fusion at the input level, merging the aligned traditional image and event representation in the channel dimension to form a unified multi-channel input tensor, which is directly fed into the backbone of the YOLO series network; mid-to-late stage fusion adopts two branches to extract features separately, and performs fusion at the network neck (FPN / PAFPN) or detection head, and can introduce channel attention or learnable weights to improve the effective channel contribution. During the training phase, geometric data augmentation (scaling, flipping, affine, etc.) is applied simultaneously on both modalities to keep the alignment intact; color augmentation (brightness, contrast, color perturbation) is limited to the traditional image branch to avoid affecting the temporal semantics of the event representation. Labels are shared in a unified, aligned coordinate system. It is recommended to collect samples covering different reflectivity, surface roughness, and dynamic perturbations to enhance generalization. During the inference phase, the fusion pipeline should be kept consistent, and temporal filtering should be added if necessary to suppress occasional false detections. To improve sensitivity to high reflectivity and weak texture defects, a lightweight attention module can be added to the neck of the network, allowing it to automatically focus on channels and scales with higher information content. For loss configuration, a combination of CIoU / DFL and classification and confidence losses should be used, with appropriate weighting according to modal contribution to obtain a more robust learning signal. Furthermore, the YOLO series network undertakes end-to-end inference from fused input to defect localization and classification. To adapt to bimodal data, the network introduces channel expansion and alignment preservation mechanisms in the input or feature layers: in early fusion, the input is merged in the channel dimension while maintaining the same input scale and alignment parameters as in the spatial alignment stage, allowing the backbone to learn static texture and dynamic contours simultaneously in the first convolutional layer; in mid-to-late fusion, a dual-branch shared or partially shared backbone is used to extract features from both modalities, and then cross-scale fusion is performed at the neck layer through lateral + top-down / bottom-up paths and channel attention, or weighted stitching is performed before the detection head to alleviate feature misalignment caused by differences in statistical distributions of different modalities. During the training phase, geometric enhancement is strictly maintained to be consistent between the two modalities, and color enhancement is restricted to the traditional image branch. Labels are uniquely shared in the aligned coordinate system. The loss uses a combination of CIoU / DFL with classification and confidence. For bimodal channels, a light weight balancing or distillation term can be set to stabilize convergence; the fusion site and scale strategy in the inference phase are consistent with those in training, and temporal smoothing or cross-frame NMS can be used to suppress occasional false detections.Regarding the detection mechanisms for different defect types, scratch-like defects appear as elongated, low-contrast structures in traditional images, but in event representation, they exhibit continuous event trajectories or edge enhancements along the direction of motion. The network captures these elongated structures through convolutional kernel orientation sensitivity and multi-scale fusion. Micro-dimples and pinholes appear as local dark spots or reflection anomalies in traditional images, but in event representation, they form local event clusters due to brightness perturbations. After fusion, the recall rate for small targets is significantly improved. Oxide spots and coating anomalies are usually accompanied by changes in reflectivity and weakened edge textures. The event channel is sensitive to boundary dynamics and micro-perturbations, providing structural clues for texture degradation areas in traditional images. On highly reflective samples, traditional images may lose texture due to local saturation, while the event channel can still generate usable events under micro-fluctuations in light intensity. The complementarity of these two approaches can improve the detection capability and localization accuracy of small defects without increasing exposure costs.

[0037] To ensure the stability of the engineering deployment, consistency and quality control were implemented in the data flow during system deployment: First, alignment consistency was verified by sampling representative images / batch processing parameters. If geometric mismatches in the fused input were found (e.g., the overlap of the reference rectangle was below the threshold), a rollback or recalibration prompt for the alignment parameters was triggered. Second, key metadata (alignment parameter version, model version, threshold setting, trigger frequency, and timestamp statistics) was recorded during the detection process to facilitate batch traceability and anomaly analysis. Third, under production cycle conditions, channel validity discrimination and adaptive gating were added for extreme cases such as event silence or traditional image overexposure to prevent the forced injection of low-quality modalities into the network. Finally, a confidence-IoU dual threshold strategy and process logic constraints (such as cross-consistency with upstream metrology or downstream appearance verification) were adopted for the output results, and retraining and threshold recalibration were performed using small batches of incremental data during sample drift to maintain long-term robustness.

[0038] In terms of performance metrics, bimodal fusion demonstrates quantifiable improvements in recall and mean average precision (mAP) compared to monomodal fusion, particularly in scenarios involving small targets (such as pinholes and micro-dimples) and areas with damaged details in highly reflective areas, where recall is significantly improved. Precision remains the same or slightly improves after incorporating temporal smoothing and channel attention. Since fusion requires strict spatiotemporal alignment, false positives are significantly reduced, the proportion of residual conflict boxes decreases after NMS, and model convergence is more stable. In practical deployments, by combining production line yield and return-to-base kernel data, mAP@0.5 and mAP@[0.5:0.95], AP for small targets, and AP for reflective areas can be monitored separately, and control charts can be established on a batch-by-batch basis to promptly identify drift and degradation.

[0039] The present invention also provides a computer device comprising the microscopic spectroscopic event camera and conventional camera alignment system described in the first aspect.

[0040] The present invention also provides a non-transitory computer-readable storage medium for storing the operating program of a microscopic spectroscopic event camera and a conventional camera alignment system.

[0041] This invention constructs a closed-loop system of "optical consistency - time synchronization - spatial alignment - fusion detection," achieving microsecond-level time synchronization through hardware triggering and pixel-level spatial consistency through two-stage spatial alignment. It then accurately fuses dual-modal data and inputs it into the YOLO series network for defect detection. This effectively solves the shortcomings of traditional frame cameras and event cameras in microscopic inspection scenarios, improves defect detection efficiency under conditions of strong reflection and rapid transients, and has good industrial application value.

[0042] It is worth noting that the terms "first" and "second" in this invention are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0043] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A microscopic spectroscopic event camera and a conventional camera alignment system, characterized in that, include: The microscopic optical path module is used to simultaneously guide the same light beam to both the event camera and the conventional camera according to a set beam splitting ratio. The hardware trigger control module is used to generate external trigger pulses and connect them to the event camera and the traditional camera respectively, so that the acquisition start point of both is locked at the same pulse trigger time. The time synchronization management module is used to perform link delay compensation on event cameras and traditional cameras respectively, using a unified timestamp as the time reference, so that the data generated by both at the same time has the same timestamp. The data acquisition module is used to acquire the event map of the event camera and the image frame of the traditional camera in parallel under the unified external trigger pulse and the time reference, and store the image frame and the event map into the corresponding cache queue; The spatial alignment module is used to align the image frames of the traditional camera with the event map of the event camera by using the relevant parameters of the event camera as a target reference and processing them through affine transformation and intelligent cropping, so as to obtain the aligned image frames and event map. The fusion detection module is used to feed the aligned image frames and event graphs into the fusion detection network for fusion detection to obtain the detection results.

2. The alignment system between the microscopic spectroscopic event camera and a conventional camera according to claim 1, characterized in that, The microscopic optical path module includes a microscope lens and a beam splitter. The microscope lens, in conjunction with the beam splitter, guides the same light beam to both the event camera and the conventional camera simultaneously according to a set splitting ratio.

3. The alignment system between a microscopic spectroscopic event camera and a conventional camera according to claim 1, characterized in that, The hardware trigger control module includes a serial communication unit, a trigger control unit, and a trigger parameter configuration unit. The serial communication unit is used to communicate with external modules; the trigger parameter configuration unit is used to set the trigger frequency, start command, and stop command; the trigger control unit is used to generate external trigger pulses according to the trigger frequency under the start command, and to stop generating the external trigger pulses under the stop command.

4. The alignment system between a microscopic spectroscopic event camera and a conventional camera according to claim 1, characterized in that, The time synchronization management module includes a hardware-triggered timestamp recording unit, an offset calibration unit, and a statistical analysis unit; The hardware trigger timestamp recording unit is used to record the trigger time of the external trigger pulse; The offset calibration unit is used to set adjustable offsets for compensating for link delay for the event camera and the traditional camera respectively, and to calibrate the event camera and the traditional camera according to the adjustable offsets. The statistical analysis module is used to count the trigger count, the average interval between trigger times, and the most recent trigger time, and to continuously monitor these three factors. When the hardware trigger control module fails to function, an external trigger pulse is generated using external software.

5. The alignment system between a microscopic spectroscopic event camera and a conventional camera according to claim 1, characterized in that, The data acquisition module includes a data acquisition subunit, an event representation construction subunit, and a preprocessing subunit; The data acquisition subunit is used to acquire the event stream of the event camera and the initial image frames of the traditional camera in parallel under a unified external trigger pulse and the time reference. The event representation construction sub-unit is used to assemble the event flow into an initial event graph; The preprocessing subunit is used to perform channel enhancement and adaptive histogram processing on the initial image frame, then perform Otsu thresholding and morphological opening and closing to stabilize the contour to obtain the image frame, and sequentially perform grayscale, Gaussian smoothing and fixed threshold binarization processing on the initial event map, and cooperate with morphological opening and closing to obtain the event map; and store the image frame and event map into the corresponding cache queue.

6. The alignment system between a microscopic spectroscopic event camera and a conventional camera according to claim 1, characterized in that, The spatial alignment module includes: a coarse alignment submodule and a fine alignment submodule; The coarse alignment submodule is used to estimate the initial scaling range of the conventional camera relative to the event camera, using the field of view, pixel size, and resolution of the event camera as a target reference. The precision alignment module is used to calculate the angle difference, uniform scale, and center translation relative to the target reference and within the initial estimation range using a rotating rectangle detection and median robust estimation method; and to perform an affine transformation on the image frame according to the angle difference, uniform scale, and center translation, and to perform intelligent cropping and mirror boundary padding to obtain the aligned image frame and event map.

7. The alignment system between a microscopic spectroscopic event camera and a conventional camera according to claim 6, characterized in that, The precision alignment module is used to extract the contours of the event graph and the aligned image frame and fit them with the minimum bounding rectangle, filter out contours with abnormal aspect ratios to obtain the rotation rectangles corresponding to the event graph and the image frame respectively; calculate multiple angle differences between the rotation matrices corresponding to the event graph and the image frame respectively, and select the median of the multiple angle differences; select the event graph and image frame that are closest to the median as the representative frame pair. Based on the angle difference of the representative frame pair and the calculated uniform scale and translation vector, all image frames are rotated, scaled and translated accordingly. Then, all translated image frames are intelligently cropped and mirrored to obtain aligned image frames and event graphs.

8. The alignment system between a microscopic spectroscopic event camera and a conventional camera according to claim 1, characterized in that, The fusion detection module includes a dual-modal fusion module and a defect detection module; The dual-modal fusion module is used to fuse the event graph with the aligned image frame to obtain a fusion tensor for detection; The defect detection module is used to input the fused tensor into the detection head to obtain the defect detection result. The defect detection result includes: a defect box marking the defect location, the defect category, and the confidence level of the corresponding defect category.

9. A computer device comprising a microscopic spectroscopic event camera and a conventional camera alignment system as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium for storing the operating program of the microscopic spectroscopic event camera and conventional camera alignment system according to any one of claims 1 to 8.

Images