Joint data acquisition method based on RGB camera and pulse camera

Abstract

The invention relates to a joint data acquisition method based on an RGB camera and a pulse camera, and belongs to the technical field of computer vision, multi-modal image acquisition and sensing fusion. The method comprises the steps of camera calibration, view field overlapping optimization calculation and camera position adjustment, double-camera data acquisition based on time synchronization, pulse data processing and data storage. Through view field overlapping calculation and optimization, the installation positions and adjustment amounts of the two cameras are accurately determined, and the maximum overlap of the view fields of the two cameras is ensured, so that the accuracy of data acquisition and the alignment quality of multi-modal data are improved; by adopting a uniform timestamp mechanism, time synchronization of data acquisition of the pulse camera and the RGB camera is realized, and the flexibility and expandability of the system are improved; according to the method, pulse data are converted into gray level images, the problem of alignment of pulse signals and RGB images is solved based on alignment of timestamps, and high-precision data support is provided for subsequent tasks such as multi-modal fusion and three-dimensional reconstruction; the method has a wide application scene.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, multimodal image acquisition and sensor fusion, and particularly to a joint data acquisition method based on an RGB camera and a pulse camera. Background Technology

[0002] A pulse camera [1] is a novel visual sensor that can output pulse signals only when pixel brightness changes through asynchronous sampling. Compared with traditional frame-based RGB cameras, pulse cameras have significant advantages, especially when dealing with high-speed moving objects. Pulse cameras can avoid motion blur problems in high-speed moving scenes by providing microsecond-level high temporal resolution and low data redundancy, and exhibit better stability and accuracy in high dynamic range environments. This makes pulse cameras particularly suitable for tasks such as dynamic scene perception and 3D reconstruction.

[0003] However, most of the existing research and datasets on pulse cameras[2] focus on scenes of high-speed moving objects using stationary cameras. Almost all existing datasets assume that the camera is stationary and mainly focus on the movement of a single object. This limitation makes it impossible to effectively apply pulse cameras in scenes involving camera movement, especially in dynamic scenes where multi-view collaboration is required or the camera itself is moving. Existing pulse camera datasets cannot meet the needs of these tasks. In contrast, RGB cameras have relatively mature technology in handling dynamic scenes involving camera movement and can provide rich image information. However, when facing high-speed moving objects, they often encounter problems such as motion blur, which affects their performance in scenes with fast-moving objects. Pulse cameras can make up for these shortcomings with their high temporal resolution. Therefore, how to combine pulse cameras with RGB cameras and use their respective advantages to collect data synchronously is an urgent problem to be solved in the current technology.

[0004] Currently, there is no effective solution for combining pulse cameras and RGB cameras for joint data acquisition, especially in dynamic scenes involving camera movement or fast-moving objects. This prevents the full utilization of the advantages of pulse cameras. Furthermore, due to differences in their working principles, data output formats, and temporal resolutions, accurately synchronizing and effectively fusing their data remains an unsolved technical challenge. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention proposes a joint data acquisition method based on an RGB camera and a pulse camera. Through precise field-of-view overlap optimization and time synchronization mechanism, synchronous acquisition by the pulse camera and the RGB camera is achieved. The acquired data can be precisely aligned in time during the acquisition process, solving the problem of time alignment between pulse data and RGB images, and providing high-precision and reliable data support for subsequent tasks such as multimodal data fusion.

[0006] The technical solution of the present invention is as follows: A joint data acquisition method based on an RGB camera and a pulse camera is characterized in that the system using this method for data acquisition includes a pulse camera, an RGB camera, an adjustable camera mounting bracket, and a data acquisition computing terminal. The pulse camera and the RGB camera are respectively mounted on the adjustable camera mounting bracket. The bracket includes a base distance adjustment structure for adjusting the relative position of the two cameras and a rotation angle adjustment structure for fine-tuning the optical axis direction, ensuring that the two cameras maintain a stable and adjustable relative pose relationship during data acquisition. Both the pulse camera and the RGB camera are connected to the computer of the data acquisition computing terminal via independent data cables for data reading, timestamp recording, and storage control of image signals and pulse signals. The method includes the following steps: (1) Camera calibration; The calibration process obtains the intrinsic parameter matrices, distortion parameters, and their values ​​relative to the calibration reference for both the pulse camera and the RGB camera. The extrinsic matrix of the coordinate system; (2) Field of view overlap optimization calculation and camera position adjustment; Using the intrinsic and extrinsic parameters obtained from camera calibration, the overlapping area of ​​the fields of view of the two cameras is calculated and optimized. Based on the calculated base distance adjustment and angle adjustment, the installation position and angle of the cameras are adjusted to maximize the overlap of the fields of view of the two cameras. (3) Data acquisition based on time synchronization of dual cameras; The system enters the data acquisition phase. The computer in the data acquisition computing terminal configures the parameters of the two cameras through acquisition software or acquisition code, and designs software scripts to control the data acquisition process of the two cameras. The acquisition status is monitored in real time by the computer's timer. A unified time base is used to calibrate the clocks of the pulse camera and the RGB camera to ensure that the start and end times of acquisition for the two cameras are completely consistent. After the acquisition begins, whenever the RGB camera acquires a frame of data, the script generates a precise timestamp for that frame, records it in the data file, and marks the pulse data acquired in the corresponding time period. When a time deviation is detected between the pulse data and the RGB image, the script automatically adjusts the buffer update and data pairing strategy of the two data streams during the acquisition process according to the frame rate of the two cameras to ensure that the timestamps of the pulse data and the RGB image are precisely aligned and always kept synchronized. After the data acquisition is completed, the acquisition process of both cameras is stopped simultaneously, and all acquired data is saved in a timestamp-aligned manner. The pulse camera acquires pulse data, and the RGB camera acquires image data. During the data acquisition process, the entire acquisition system is moved directly to collect data. (4) Pulse data processing; The pulse data acquired by the pulse camera is processed to reconstruct textures and converted into a grayscale image sequence. The converted grayscale images are then saved according to the timestamp alignment of the pulse data.

[0007] Furthermore, in step (2), using Represents a pulse camera, using Representing RGB cameras, specifically including: First, using the intrinsic parameter matrices of the two cameras, the horizontal field of view of each camera is calculated. and vertical field of view ; Next, using the extrinsic matrices of the two cameras, the optical axis directions of the two cameras are calculated, and the angle representing the directional difference between the two cameras is obtained. ,in, and Let the optical axis direction vectors of the two cameras be used to calculate the angle adjustment: , and These are the horizontal field of view angles of the two cameras; the adjustment amount of this angle... It is used to guide users to rotate the angle adjustment structure on the bracket according to the adjustment results; After adjusting the angle, calculate the overlap area of ​​the fields of view of the two cameras; first calculate the base distance between the two cameras. and with minimum overlap width Comparison: The base distance adjustment amount is obtained. To ensure that the field of view of the two cameras is within the working distance The base distance can be fully overlapped; the adjustment amount is based on this base distance. Fine-tuning is performed by adjusting the base distance adjustment structure on the bracket to ensure that the horizontal base distance of the two cameras meets the requirements of optimal field of view overlap.

[0008] Furthermore, step (4) involves texture reconstruction processing of the pulse data using the pulse interval-based texture reconstruction (TFI) method, specifically as follows: Set at pixel The pulse stream recorded at the location is ,in For the pulse timestamp, To indicate the polarity of the brightness change, the time interval between adjacent pulses is defined as... According to the logarithmic intensity variation model of the pulse camera, we have: ,in As the trigger threshold, For pixels at time Light intensity, TFI method utilizes pulse interval Approximate recovery of grayscale information assumes that grayscale intensity is inversely proportional to the time interval, i.e. This indicates that the denser the pulse signal, the more significant the pixel texture details; further, a normalization formula is used to obtain standardized grayscale values: ,in and These represent the maximum and minimum pulse intervals within the time window, respectively, and the resulting grayscale values ​​are mapped to... or Within a certain range, grayscale image reconstruction was achieved.

[0009] Furthermore, the acquired image data, pulse data, and grayscale images converted from pulse data are saved using a timestamp-aligned method to form a structured multimodal dataset. The specific storage mechanism is as follows: After data acquisition is completed, the computer generates a unique folder based on the acquisition time of each acquisition. This folder contains pulse data from the pulse camera, image data acquired by the RGB camera, and the reconstructed grayscale image corresponding to the pulse data. Each folder is named according to the acquisition timestamp. Pulse data, grayscale images, and RGB images are stored in different subfolders. The computer timestamps each frame of RGB image, its corresponding pulse data, and its reconstructed grayscale image, and generates an index file that records the correspondence between each frame of RGB image, its corresponding pulse data, and its reconstructed image. This index file is stored in the data storage directory.

[0010] The technical effects of this invention are as follows: This invention relates to a joint data acquisition method based on an RGB camera and a pulse camera. Through field-of-view overlap calculation and optimization, it is able to... Precisely determining the installation positions and adjustment amounts of the two cameras ensures maximum overlap of their fields of view, thereby improving the accuracy of data acquisition and the alignment quality of multimodal data. By employing a unified timestamp mechanism, time synchronization of data acquired by the pulse camera and RGB camera can be achieved, enhancing the system's flexibility and scalability. Converting pulse data to grayscale images and aligning them based on timestamps solves the alignment problem between pulse signals and RGB images, providing high-precision data support for subsequent tasks such as multimodal fusion and 3D reconstruction. This invention fully leverages the advantages of both cameras and is effectively applied to scenarios involving fast-moving objects and camera motion itself. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the overall structure of the data acquisition system based on the combined RGB camera and pulse camera of the present invention; Figure 2 This is a flowchart of the data acquisition method based on the combined RGB camera and pulse camera of the present invention. Detailed Implementation

[0012] The present invention will be further clearly and completely described below with reference to the accompanying drawings and specific embodiments.

[0013] This invention discloses a joint data acquisition method based on an RGB camera and a pulse camera. Through mechanisms such as field-of-view overlap optimization and time synchronization, it achieves efficient and stable joint acquisition of data from the RGB camera and the pulse camera within a unified field of view. First, the RGB camera and the pulse camera are calibrated by capturing a checkerboard image on a calibration board, obtaining the intrinsic parameter matrices of the two cameras and their extrinsic parameters, thereby determining the relative pose of the cameras. Based on the calibration parameters, a field-of-view model of the two cameras is established, and the field-of-view overlap area of ​​the two cameras in the current installation state is calculated. Based on the calculation results of the field-of-view overlap, the optimal installation position of the cameras is obtained. Then, the base distance and optical axis angle of the camera support are adjusted to maximize the field-of-view overlap of the two cameras, completing the installation of the acquisition system device.

[0014] After the acquisition system is installed, it simultaneously initiates the data acquisition process of both cameras via the computer on the data acquisition computing terminal. During acquisition, the entire acquisition system can be moved directly to collect data. The RGB camera and pulse camera output image frames and pulse signals respectively. The computer associates the data streams of both cameras using timestamps, ensuring that the acquired pulse data is aligned with the RGB image in terms of timestamps. Subsequently, the pulse data undergoes reconstruction and other processing to reconstruct a grayscale image aligned with the RGB image. The computer stores all acquired multimodal data by timestamp association, forming a structured multimodal dataset that provides reliable data support for subsequent multimodal fusion tasks.

[0015] like Figure 1 As shown, the joint data acquisition system using the method of this invention mainly consists of a pulse camera, an RGB camera, an adjustable camera mounting bracket, and a data acquisition computing terminal. The pulse camera and the RGB camera are respectively mounted on the camera mounting bracket. The bracket includes a base distance adjustment structure for adjusting the relative position of the two cameras and a rotation angle adjustment structure for fine-tuning the optical axis direction, ensuring that the two cameras maintain a stable and adjustable relative pose relationship during data acquisition. Both the pulse camera and the RGB camera are connected to the computer (such as a laptop) of the data acquisition computing terminal via independent data cables for data reading, timestamp recording, and storage control of image signals and pulse signals.

[0016] like Figure 2 As shown, the data acquisition method of this invention includes multiple steps, including camera calibration, field-of-view overlap optimization calculation of the relative positions of the cameras, camera position adjustment, dual-camera data acquisition, pulse data processing, and data storage. First, the intrinsic and extrinsic parameters of the pulse camera and the RGB camera are obtained through the camera calibration process. Then, based on the parameters obtained from the calibration, a field-of-view model of the two cameras is established, and their field-of-view overlap is calculated. The position of the camera mounting bracket is adjusted according to the overlap result to ensure the two cameras reach the optimal field-of-view overlap area. After the dual-camera position adjustment is completed, the system enters the formal acquisition phase, directly acquiring data through a mobile acquisition system. The computer of the data acquisition computing terminal synchronously starts data acquisition from both cameras, and time synchronization of the acquisition process is achieved through a unified system timestamp. After the acquisition cycle ends, the computer of the data acquisition computing terminal writes the two types of data to a file according to the time index and reconstructs the pulse data to form a time-aligned multimodal dataset. The various steps of this invention will be described in detail below.

[0017] (I) Camera Calibration In this invention, the camera calibration step is used to obtain the intrinsic parameter matrices, distortion parameters, and extrinsic parameter matrices (rotation matrix and translation vector) of the pulse camera and the RGB camera relative to the calibration reference coordinate system, respectively. During the calibration process, multiple frames containing a checkerboard pattern are acquired separately for each camera (the pulse camera obtains an equivalent grayscale frame through reconstruction), and the camera's projection model is calculated using the calibration method. For any given camera, its intrinsic parameter matrix is... The form is: in, and It's the camera's focal length. and These are the pixel coordinates of the optical center. During calibration, the corner points in the three-dimensional world coordinate system are used. Corner positions in a two-dimensional image coordinate system The intrinsic parameters are solved by considering the relationships between points. Using a camera imaging model, 3D points can be... Projecting onto a two-dimensional image plane yields image points. : in, It is the camera's rotation matrix, describing the rotation relationship between the camera coordinate system and the world coordinate system. It is a translation vector, representing the translation distance from the origin of the camera coordinate system to the origin of the world coordinate system. The two together form the extrinsic parameter matrix. In solving the above projection relationship, distortion parameters (including radial and tangential distortion coefficients) are also solved as optimization variables. The calibration algorithm first needs to ideally project the 3D points based on the currently estimated intrinsic and extrinsic parameters, and then apply the distortion model to the projected point positions to make them more consistent with the corner positions on the real image. By performing nonlinear least squares optimization on the reprojection errors of all corner points in multiple frames of images, the camera intrinsic parameter matrix can be obtained simultaneously. Distortion parameters, rotation matrix Translation vector This allows us to obtain the complete calibration parameters for both cameras. From this, we obtain the intrinsic and extrinsic parameters of the two cameras.

[0018] (II) Field of view overlap optimization calculation and camera position adjustment After completing the camera calibration and obtaining the corresponding intrinsic and extrinsic parameters, the goal of this step is to calculate and optimize the overlapping area of ​​the fields of view of the two cameras, and adjust the installation position and angle of the cameras according to the calculation results to ensure that the fields of view of the two cameras have sufficient overlap, thereby meeting the needs of subsequent data fusion.

[0019] Here it is used Represents a pulse camera, using This represents an RGB camera. First, the intrinsic parameter matrices of the two cameras are used. and Calculate the field of view of each camera ( According to the camera's focal length. and and image size The horizontal field of view can be calculated separately. and vertical field of view : Next, the extrinsic parameter matrices of the two cameras are used. and Calculate the optical axis directions of the two cameras, and obtain the included angle from them. , indicating the directional difference between the two cameras, where, and These are the optical axis direction vectors of the two cameras, respectively. Adjustment amount Given by the following formula: This adjustment amount can be used to guide users to rotate the camera mount according to the adjustment results, thereby optimizing the field of view overlap between cameras. and These are the horizontal field of view angles of the two cameras, respectively.

[0020] After adjusting the angle, this step further calculates the overlap area of ​​the fields of view of the two cameras. First, the base distance between the two cameras is calculated. and with minimum overlap width Comparison: This allows us to obtain the base distance adjustment. To ensure that the field of view of the two cameras is within the working distance The two cameras can be fully overlapped. Users can fine-tune the adjustment by using the slide rails on the bracket, ensuring the horizontal base distance between the two cameras meets the requirements for optimal field-of-view overlap. Through these steps, the system can calculate appropriate angle and base distance adjustments to maximize the overlap of the two cameras' fields of view, thus ensuring spatial consistency during data acquisition. Users can then adjust the positions of the two cameras to complete the installation of the entire acquisition system.

[0021] (III) Time-synchronized dual-camera data acquisition After optimizing the field of view overlap and adjusting the camera's mounting position, the system enters the data acquisition phase. The core task of this phase is to ensure that all data acquired by the pulse camera and the RGB camera during the acquisition process have consistent timestamps. The data streams of the two cameras are correlated through timestamps, and the pulse data and RGB images are aligned in timestamp after acquisition, thereby providing an accurate time reference for subsequent data applications.

[0022] The pulse camera and the RGB camera are connected to the computer of the data acquisition and computing terminal through their respective interfaces. The pulse camera is configured using the dedicated acquisition software SpikeSee ​​to set appropriate parameters such as sampling frequency and exposure. The RGB camera is configured using general acquisition code to set parameters such as frame rate and resolution for image acquisition.

[0023] During data acquisition, a software script was designed to control the data acquisition from both cameras. This script, located on the computer of the data acquisition computing terminal, controls the acquisition process of the two cameras, ensuring precise synchronization between the start and end times of data acquisition from the pulse camera and the RGB camera. It also ensures that each pulse data segment acquired by the pulse camera is precisely aligned with each frame of image acquired by the RGB camera in terms of timestamps. To guarantee time consistency, a computer timer monitors the acquisition status in real time, and a unified time base is used to calibrate the clocks of both the pulse camera and the RGB camera, ensuring that the start and end times of acquisition for both cameras are completely consistent. After acquisition begins, whenever the RGB camera acquires a frame of data, the script generates a precise timestamp for that frame, records it in the data file, and marks the pulse data acquired in the corresponding time period. When a time discrepancy is detected between the pulse data and the RGB image, the script automatically adjusts the buffer update and data pairing strategy for both data streams during acquisition based on the frame rates of the two cameras, ensuring precise alignment of the timestamps of the pulse data and the RGB image, maintaining constant synchronization.

[0024] After data acquisition is completed, the computer simultaneously stops the acquisition process of both cameras and saves all acquired data. Each piece of acquired data is accompanied by its corresponding timestamp to ensure time alignment between the pulse data and the RGB image, facilitating subsequent data processing, pulse data reconstruction, and multimodal data fusion tasks.

[0025] Through this time synchronization mechanism, the present invention achieves precise time synchronization of data acquisition between the pulse camera and the RGB camera, ensuring high-precision data alignment and providing reliable data support for subsequent tasks such as multimodal data fusion.

[0026] (iv) Pulse Data Processing After the pulse camera completes data acquisition, the raw output obtained directly is "Raw Spikes", which is the pulse stream data generated by the camera. Each pulse records the spatial position, timestamp, and brightness change polarity of the trigger pixel. Since this data format is different from the frame-by-frame grayscale image of the traditional camera, it cannot be directly used for calibration and three-dimensional reconstruction tasks. Therefore, it must be converted into a usable grayscale image sequence through texture reconstruction processing. To this end, the present invention adopts the ultra-high speed motion scene texture reconstruction algorithm, namely the pulse interval-based texture reconstruction (TFI) method [3][4], to restore the complete texture of the natural scene with microsecond-level time resolution, avoiding the serious motion blur problem caused by the traditional camera in high-speed motion scenes. When the converted grayscale image data is saved, it is saved in the pulse data timestamp alignment mode.

[0027] Specifically, set at pixels The pulse stream recorded at the location is ,in For the pulse timestamp, To indicate the polarity of the brightness change, the time interval between adjacent pulses can be defined as... According to the logarithmic intensity variation model of the pulse camera, we have: in As the trigger threshold, For pixels at time The light intensity. The TFI method utilizes the pulse interval. To approximate the recovery of grayscale information, it is generally assumed that grayscale intensity is inversely proportional to the time interval, i.e. This indicates that the denser the pulse signal, the more significant the pixel texture details. To obtain standardized grayscale values, a normalization formula is further employed: in and These represent the maximum and minimum pulse intervals within the time window, respectively, and the resulting grayscale values ​​are mapped to... Within the range of [0, 1], effective reconstruction of grayscale images was achieved.

[0028] In the experimental setup of this invention, the acquisition duration of each pulse sequence is 3 seconds, generating an average of approximately 60,000 pulse frames. Direct processing would result in excessive data volume and computational redundancy. To improve efficiency, a method is adopted... The pulse frame compression strategy synthesizes 10 consecutive pulse frames into a single grayscale image, resulting in approximately 6,000 reconstructed images per sequence. This process effectively converts the original pulse stream data into a grayscale image sequence, preserving the high temporal resolution advantage of the pulse camera in high-speed dynamic scenes while significantly reducing data redundancy, thus providing a high-quality data foundation for subsequent tasks.

[0029] To ensure efficient and secure storage of multimodal data after acquisition, and convenient access and use in subsequent processing stages, this invention provides a standardized storage format to ensure data traceability and availability. After data acquisition, the computer generates a unique folder based on the acquisition time of each acquisition. This folder contains pulse data from the pulse camera, image data acquired by the RGB camera, and the reconstructed grayscale image corresponding to the pulse data. Each folder is named according to the acquisition timestamp for easy management and retrieval. Pulse data, grayscale images, and RGB images are stored in different subfolders for easy classification, management, and retrieval. To ensure the synchronization of pulse data and RGB images, the computer timestamps each frame of RGB image, its corresponding pulse data, and its reconstructed grayscale image, and generates an index file recording the correspondence between each frame of RGB image, its corresponding pulse data, and its reconstructed image. This index file is stored in the data storage directory for easy data retrieval and alignment.

[0030] Through this storage mechanism, the present invention ensures the efficiency and consistency of the collected multimodal data in terms of format, and through structured management, ensures that the subsequent processing, analysis and application of multimodal data can be carried out in an efficient and stable environment.

[0031] The method of this invention can accurately synchronize data from different types of sensors and can be widely used in tasks such as 3D reconstruction and dynamic scene perception.

[0032] [1]HuangT, Zheng Y, Yu Z, et al. 1000× faster camera and machinevision with ordinary devices[J]. Engineering, 2023, 25: 110-119. [2]Zheng Y, Zheng L, Yu Z, et al. High-speed image reconstruction through short-term plasticity for spiking cameras[C] / / Proceedings of the IEEE / CVF Conference onComputerVision and Pattern Recognition. 2021: 6358-6367 [3]Yang S, Huang Z, Chang Y, et al. Real-Data-Driven 2000 FPS ColorVideofrom Mosaicked Chromatic Spikes[C] / / European Conference on ComputerVision.Cham: Springer Nature Switzerland, 2024: 305-321. [4]Zhu L, Dong S, Huang T, et al. A retina-inspired sampling methodforvisual texture reconstruction[C] / / 2019 IEEE International ConferenceonMultimedia and Expo (ICME). IEEE, 2019: 1432-1437.

Claims

1. A method for joint data acquisition based on an RGB camera and a pulse camera, characterized in that, The system for data acquisition using this method includes a pulse camera, an RGB camera, an adjustable camera mount, and a data acquisition computing terminal. The pulse camera and the RGB camera are respectively mounted on the adjustable camera mount. The mount includes a base distance adjustment structure for adjusting the relative position of the two cameras and a rotation angle adjustment structure for fine-tuning the optical axis direction, ensuring that the two cameras maintain a stable and adjustable relative pose during data acquisition. Both the pulse camera and the RGB camera are connected to the computer in the data acquisition computing terminal via independent data cables for data reading, timestamp recording, and storage control of image and pulse signals. The method includes the following steps: (1) Camera calibration; The calibration process obtains the intrinsic parameter matrices, distortion parameters, and their values ​​relative to the calibration reference for both the pulse camera and the RGB camera. The extrinsic matrix of the coordinate system; (2) Field of view overlap optimization calculation and camera position adjustment; Using the intrinsic and extrinsic parameters obtained from camera calibration, the overlapping area of ​​the fields of view of the two cameras is calculated and optimized. Based on the calculated base distance adjustment and angle adjustment, the installation position and angle of the cameras are adjusted to maximize the overlap of the fields of view of the two cameras. (3) Data acquisition based on time synchronization of dual cameras; The system enters the data acquisition phase. The computer in the data acquisition computing terminal configures the parameters of the two cameras through acquisition software or acquisition code, and designs software scripts to control the data acquisition process of the two cameras. The acquisition status is monitored in real time by the computer's timer. A unified time base is used to calibrate the clocks of the pulse camera and the RGB camera to ensure that the start and end times of acquisition for the two cameras are completely consistent. After the acquisition begins, whenever the RGB camera acquires a frame of data, the script generates a precise timestamp for that frame, records it in the data file, and marks the pulse data acquired in the corresponding time period. When a time deviation is detected between the pulse data and the RGB image, the script automatically adjusts the buffer update and data pairing strategy of the two data streams during the acquisition process according to the frame rate of the two cameras to ensure that the timestamps of the pulse data and the RGB image are precisely aligned and always kept synchronized. After the data acquisition is completed, the acquisition process of both cameras is stopped simultaneously, and all acquired data is saved in a timestamp-aligned manner. The pulse camera acquires pulse data, and the RGB camera acquires image data. During the data acquisition process, the entire acquisition system is moved directly to collect data. (4) Pulse data processing; The pulse data acquired by the pulse camera is processed to reconstruct textures and converted into a grayscale image sequence. The converted grayscale images are then saved according to the timestamp alignment of the pulse data.

2. The method as described in claim 1, characterized in that, The step (2) uses Represents a pulse camera, using Representing RGB cameras, specifically including: First, using the intrinsic parameter matrices of the two cameras, the horizontal field of view of each camera is calculated. and vertical field of view ; Next, using the extrinsic parameters of the two cameras, the optical axis directions of the two cameras are calculated, and from this, the relationship between the two cameras is obtained. The angle between the directions ,in, and Let the optical axis direction vectors of the two cameras be used to calculate the angle adjustment: , and These are the horizontal field of view angles of the two cameras; the adjustment amount of this angle... It is used to guide users to rotate the angle adjustment structure on the bracket according to the adjustment results; After adjusting the angle, calculate the overlap area of ​​the fields of view of the two cameras; first calculate the base distance between the two cameras. and with minimum overlap width Comparison: The base distance adjustment amount is obtained. To ensure that the field of view of the two cameras is within the working distance The base distance can be fully overlapped; the adjustment amount is based on this base distance. Fine-tuning is performed by adjusting the base distance adjustment structure on the bracket to ensure that the horizontal base distance of the two cameras meets the requirements of optimal field of view overlap.

3. The method as described in claim 1, characterized in that, Step (4) involves texture reconstruction processing of the pulse data using the pulse interval-based texture reconstruction (TFI) method, specifically: Set at pixel The pulse stream recorded at the location is ,in For the pulse timestamp, To indicate the polarity of the brightness change, the time interval between adjacent pulses is defined as... According to the logarithmic intensity variation model of the pulse camera, we have: ,in As the trigger threshold, For pixels at time Light intensity, TFI method utilizes pulse interval Approximate recovery of grayscale information assumes that grayscale intensity is inversely proportional to the time interval, i.e. This indicates that the denser the pulse signal, the more significant the pixel texture details; further, a normalization formula is used to obtain standardized grayscale values: ,in and These represent the maximum and minimum pulse intervals within the time window, respectively, and the resulting grayscale values ​​are mapped to... or Within a certain range, grayscale image reconstruction was achieved.

4. The method as described in claim 1, characterized in that, The acquired image data, pulse data, and grayscale images converted from pulse data are saved using a timestamp-aligned method to form a structured multimodal dataset. The specific storage mechanism is as follows: After data acquisition is completed, the computer generates a unique folder based on the acquisition time of each acquisition. This folder contains pulse data from the pulse camera, image data acquired by the RGB camera, and the reconstructed grayscale image corresponding to the pulse data. Each folder is named according to the acquisition timestamp. Pulse data, grayscale images, and RGB images are stored in different subfolders. The computer timestamps each frame of RGB image, its corresponding pulse data, and its reconstructed grayscale image, and generates an index file that records the correspondence between each frame of RGB image and the corresponding pulse data and reconstructed image. The index file is stored in the data storage directory.

Images