Sensor synchronization apparatus and method

Abstract

The present invention relates to a sensor synchronization apparatus and method. According to one embodiment of the present invention, a sensor synchronization apparatus provides a synchronization function of a lidar sensor and a plurality of camera sensors, thereby ensuring that fused data of accurate lidar-based point cloud data and camera-based image data is generated in real time even in a dynamic environment.

Description

Sensor synchronization processing device and method

[0001] The present invention relates to a sensor synchronization processing device and method, and more particularly, to a sensor synchronization processing technology that compensates for a delay offset between a lidar sensor used for recognizing the surrounding environment and a plurality of camera sensors.

[0002] Awareness of the surrounding environment is one of the key requirements for safety and reliability.

[0003] Typically, a perception system must continuously recognize its surroundings by processing 2D images acquired from a camera or 3D point cloud data from a LiDAR sensor.

[0004] However, each sensor has its own limitations, and these limitations can be overcome by fusing information from two sensors.

[0005] Additionally, various benchmark datasets have been released to verify this, and they generally adopt a fusion approach of lidar and camera sensors to detect the surrounding environment.

[0006] These methods are susceptible to misalignment due to the correlation between point clouds and pixels.

[0007] In the prior art, sensor synchronization has been mentioned as a crucial process for achieving high accuracy in sensor fusion.

[0008] Because lidar and camera sensors operate at different frequencies, it becomes difficult to align data from the two sensors to represent the environment at a specific time.

[0009] Existing synchronization methods may not fully meet these requirements.

[0010] In general, the currently existing synchronization methods can be classified into hardware-triggered synchronization, software synchronization, network synchronization, and clock synchronization.

[0011] Hardware synchronization is a method used to achieve precise synchronization between multiple sensors by sending an external trigger signal from a master device to a slave device.

[0012] In hardware synchronization cases, an external synchronization control device is sometimes used to send a trigger signal to multiple devices to start collecting data from the sensors.

[0013] More commonly, cameras equipped with hardware triggers use this approach to establish synchronization between sensors.

[0014] Although the hardware-level synchronization approach introduced by the authors on the KITTI benchmark dataset has proven promising, its applicability is limited to mechanical lidars that are externally rotating, making it difficult to apply to modern lidar sensors.

[0015] Additionally, externally mounted reed switches are fragile devices that can introduce potential errors or inaccuracies into the synchronization process, and they lack the flexibility to dynamically adjust for synchronization.

[0016] This can be problematic in scenarios where real-time adjustments are required to improve synchronization accuracy.

[0017] Software-based synchronization approaches offer several advantages over hardware synchronization approaches in terms of flexibility and adaptability, but they have limitations that make it difficult to meet real-time requirements in autonomous driving.

[0018] The synchronization framework implemented in the Robot Operating System (ROS) system is a well-known software-based synchronization framework and is commonly used in real-time detection applications.

[0019] Additionally, several methods have been introduced to improve the efficiency of ROS software-based synchronization.

[0020] Despite these improvements, ROS software-based synchronization is limited by latency in data transmission and processing.

[0021] Since ROS operates on a distributed system architecture, communication and data processing between different nodes can cause latency, which can affect real-time synchronization of LiDAR and camera data.

[0022] Additionally, the performance of ROS-based synchronization is highly dependent on the system's performance and network conditions, which may further impact accuracy.

[0023] Network synchronization methods typically use a centralized clock or timestamp mechanism to establish a common time reference across all devices.

[0024] We then synchronize the sensor data frames based on their associated timestamps and ensure that the sensor data from different devices are accurately correlated with each other.

[0025] More commonly, precision time protocol (PTP) and network time protocol (NTP) are used to establish network synchronization between devices.

[0026] PTP is a network protocol specifically designed for clock synchronization in distributed systems.

[0027] Sub-microsecond accuracy is achieved using hardware timestamps and synchronized message exchanges.

[0028] And it operates in a master-slave architecture where the master clock provides information to the slave clocks.

[0029] NTP, on the other hand, is a widely used protocol for synchronizing clocks over the Internet.

[0030] NTP can achieve synchronization within milliseconds or microseconds, depending on network conditions.

[0031] However, network-based synchronization depends on the availability and stability of the network infrastructure.

[0032] Network outages or fluctuations can affect synchronization accuracy and introduce inconsistencies in sensor data.

[0033] Additionally, network-based synchronization methods can achieve millisecond to sub-microsecond accuracy, but they do not provide the same level of precision as hardware-based synchronization methods.

[0034] Clock synchronization between multiple sensors refers to the process of aligning the timestamps of multi-sensor data to enable accurate sensor fusion.

[0035] This synchronization can be achieved through a variety of methods, including hardware synchronization, software synchronization, and a combination of the two.

[0036] This synchronization method allows one clock to correct its time to match another clock by synchronizing the clocks.

[0037] However, this synchronization may not apply to the clock synchronization of the lidar sensor and the camera sensor.

[0038] The purpose of the present invention is to provide a sensor synchronization processing device and method that compensates for delay offset between a lidar sensor used for recognizing the surrounding environment and a plurality of camera sensors.

[0039] The present invention aims to provide a synchronization function of a lidar sensor and a plurality of camera sensors to ensure real-time generation of fusion data of accurate lidar-based point cloud data and camera-based image data even in a dynamic environment.

[0040] The present invention aims to overcome complex delay time problems in real time depending on system environment, network load, and sensor status by providing a synchronization function of a lidar sensor and a plurality of camera sensors.

[0041] According to one embodiment of the present invention, a sensor synchronization processing device may include a data collection unit that collects point cloud data and image data using a LiDAR sensor and a plurality of camera sensors, a delay offset estimation unit that projects the point cloud data onto the image data to select a reference point and estimates a delay offset based on the selected reference point, a delay correction unit that calculates a compensation parameter that matches a reference time of the LiDAR sensor with each reference time of the plurality of camera sensors based on the estimated delay offset and corrects the shooting time of each of the plurality of camera sensors to perform delay correction, and a synchronization processing unit that synchronizes the LiDAR sensor and the plurality of camera sensors based on the performed delay correction.

[0042] The above delay offset estimation unit can estimate an average projection error and an average delay time by converting a time stamp of the point cloud data and a time stamp of the image data into a vector and projecting and mapping a 3D lidar point of the point cloud data onto an image plane using a transformation matrix.

[0043] The above delay offset estimation unit can select a reference point based on the pixel distance between the center of the camera image and the 3D lidar point by projecting and mapping the 3D lidar point of the point cloud data onto an image plane.

[0044] The above delay offset estimation unit can determine the vertical and horizontal lidar angles for the selected reference point to the lidar sensor as a starting point for starting a trigger signal for the camera sensor.

[0045] The above delay compensation unit can perform delay compensation by receiving each trigger signal from the plurality of camera sensors for the lidar sensor and correcting each shooting time to compensate for the estimated delay offset.

[0046] The above synchronization processing unit can perform time synchronization of the lidar sensor and the plurality of camera sensors by adjusting the delay time of the plurality of camera sensors determined according to the compensation parameter for a reference time difference between the sensors caused by at least one of heat generation, communication delay, and operation delay.

[0047] A sensor synchronization processing method according to an embodiment of the present invention may include a step of collecting point cloud data and image data using a LiDAR sensor and a plurality of camera sensors in a data collection unit, a step of projecting the point cloud data onto the image data to select a reference point and estimating a delay offset based on the selected reference point in a delay offset estimation unit, a step of performing delay compensation by calculating a compensation parameter that matches a reference time of the LiDAR sensor with a reference time of each of the plurality of camera sensors based on the estimated delay offset and correcting the shooting time of each of the plurality of camera sensors, and a step of performing synchronization of the LiDAR sensor and the plurality of camera sensors based on the performed delay compensation in a synchronization processing unit.

[0048] The step of collecting point cloud data and image data using the above LiDAR sensor and a plurality of camera sensors may include a step of estimating internal parameters and external parameters for the plurality of camera sensors, projecting a 3D LiDAR point of the point cloud data onto a 2D pixel of the image data to transform coordinates from a LiDAR coordinate system to an image coordinate system, and obtaining a 2D point that is an image projection of the 3D LiDAR point.

[0049] The step of projecting the point cloud data onto the image data to select a reference point and estimating a delay offset based on the selected reference point may include the step of converting a time stamp of the point cloud data and a time stamp of the image data into a vector, and using a transformation matrix to project and map a 3D lidar point of the point cloud data onto an image plane to estimate an average projection error and an average delay time.

[0050] The step of projecting the point cloud data onto the image data to select a reference point and estimating a delay offset based on the selected reference point may include the step of projecting and mapping a 3D lidar point of the point cloud data onto an image plane to select a reference point based on a pixel distance between the center of the image and the 3D lidar point, and the step of calculating vertical and horizontal lidar angles for the selected reference point to determine the selected reference point as a starting point for starting each trigger signal of the lidar sensor for the plurality of camera sensors.

[0051] The step of performing delay compensation by correcting the shooting time of each of the plurality of camera sensors may include a step of performing delay compensation by correcting the shooting time of each of the plurality of camera sensors using each trigger signal of the lidar sensor for the plurality of camera sensors to compensate for the estimated delay offset.

[0052] The step of performing synchronization of the lidar sensor and the plurality of camera sensors based on the performed delay compensation may include performing synchronization of the lidar sensor and the plurality of camera sensors based on a trigger signal of the plurality of camera sensors determined according to the compensation parameter, based on a difference between the reference time of the lidar sensor and the reference time of the plurality of camera sensors, which fluctuates due to at least one of heat generation, communication delay, and computational delay.

[0053] The present invention can provide a sensor synchronization processing device and method for compensating for delay offset between a lidar sensor used for recognizing the surrounding environment and a plurality of camera sensors.

[0054] The present invention provides a synchronization function of a lidar sensor and a plurality of camera sensors, thereby ensuring real-time generation of fusion data of accurate lidar-based point cloud data and camera-based image data even in a dynamic environment.

[0055] The present invention provides a synchronization function of a lidar sensor and multiple camera sensors, thereby overcoming complex delay time problems depending on the system environment, network load, and sensor status in real time.

[0056] FIG. 1 is a drawing illustrating a sensor synchronization processing device according to one embodiment of the present invention.

[0057] FIG. 2 and FIG. 3 are drawings explaining the application structure of a sensor synchronization processing device according to one embodiment of the present invention.

[0058] FIG. 4 is a drawing illustrating a lidar point projection of a sensor synchronization processing device according to an embodiment of the present invention.

[0059] FIG. 5 is a drawing explaining reference point estimation of a lidar scan of a sensor synchronization processing device according to an embodiment of the present invention.

[0060] FIG. 6 is a drawing illustrating a synchronization structure of a sensor synchronization processing device according to one embodiment of the present invention.

[0061] FIG. 7 is a drawing illustrating a synchronization algorithm of a sensor synchronization processing device according to one embodiment of the present invention.

[0062] FIG. 8 is a diagram illustrating projection error calculation related to synchronization performance of a sensor synchronization processing device according to an embodiment of the present invention.

[0063] FIGS. 9A to 9D are drawings explaining the synchronization performance of a sensor synchronization processing device according to one embodiment of the present invention.

[0064] FIG. 10 is a drawing explaining a sensor synchronization processing method according to one embodiment of the present invention.

[0065] Specific structural or functional descriptions of embodiments according to the concept of the present invention disclosed in this specification are merely illustrative for the purpose of explaining embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention may be implemented in various forms and are not limited to the embodiments described in this specification.

[0066] Embodiments according to the concept of the present invention may have various modifications and take various forms, and thus, embodiments are illustrated in the drawings and described in detail in this specification. However, this is not intended to limit embodiments according to the concept of the present invention to specific disclosed forms, but rather includes modifications, equivalents, or alternatives that fall within the spirit and technical scope of the present invention.

[0067] While terms such as "first" or "second" may be used to describe various components, these components should not be limited by these terms. These terms are intended solely to distinguish one component from another. For example, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component," without departing from the scope of the invention.

[0068] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Expressions that describe relationships between components, such as "between," "immediately between," or "directly adjacent to," should be interpreted similarly.

[0069] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the present invention. The singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, it should be understood that the terms "comprises" or "has" are intended to specify the presence of a described feature, number, stage, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, numbers, stages, operations, components, parts, or combinations thereof.

[0070] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0071] Hereinafter, embodiments will be described in detail with reference to the attached drawings. However, the scope of the patent application is not limited or restricted by these embodiments. The same reference numerals in each drawing represent the same components.

[0072] FIG. 1 is a drawing illustrating a sensor synchronization processing device according to one embodiment of the present invention.

[0073] FIG. 1 illustrates components of a sensor synchronization processing device according to one embodiment of the present invention.

[0074] Referring to FIG. 1, a sensor synchronization processing device (100) according to one embodiment of the present invention includes a data collection unit (110), a delay offset estimation unit (120), a delay correction unit (130), and a synchronization processing unit (140).

[0075] The data collection unit (110) can collect point cloud data and image data using a LiDAR sensor and multiple camera sensors.

[0076] The data collection unit (110) estimates internal parameters and external parameters for multiple camera sensors, projects a 3D lidar point of point cloud data onto a 2D pixel of image data, and converts coordinates from the lidar coordinate system to the image coordinate system to obtain a 2D point that is an image projection of the 3D lidar point.

[0077] The delay offset estimation unit (120) can project point cloud data onto the image data to select a reference point and estimate a delay offset based on the selected reference point.

[0078] The delay offset estimation unit (120) converts the time stamp of the point cloud data and the time stamp of the image data into vectors, and uses a transformation matrix to project and map the 3D lidar points of the point cloud data onto the image plane, thereby estimating the average projection error and average delay time.

[0079] The delay offset estimation unit (120) can select a reference point based on the distance between the center of the camera image and the 3D lidar point by projecting and mapping the 3D lidar point of the point cloud data onto the image plane.

[0080] The delay offset estimation unit (120) can calculate vertical and horizontal lidar angles for a selected reference point and determine the selected reference point as a starting point for starting each trigger signal of a lidar sensor for a plurality of camera sensors.

[0081] The delay compensation unit (130) can perform delay compensation by calculating a compensation parameter that matches the reference time of the lidar sensor and the reference time of each of the plurality of camera sensors based on the estimated delay offset, thereby correcting the shooting time of each of the plurality of camera sensors.

[0082] The delay compensation unit (130) can perform delay compensation to compensate for the estimated delay offset by correcting the shooting time of each of the plurality of camera sensors using each trigger signal of the lidar sensor for the plurality of camera sensors.

[0083] The synchronization processing unit (140) can perform synchronization of the lidar sensor and the plurality of camera sensors based on the performed delay compensation.

[0084] The synchronization processing unit (140) can perform synchronization of the lidar sensor and the plurality of camera sensors based on the delay time of the plurality of camera sensors determined according to the compensation parameter for the reference time difference that varies between the sensors due to various reasons (heat generation, communication delay, operation delay, etc.).

[0085] Accordingly, the present invention aims to provide a sensor synchronization processing device and method that compensates for delay offset between a lidar sensor used for recognizing the surrounding environment and a plurality of camera sensors.

[0086] FIG. 2 and FIG. 3 are drawings explaining the application structure of a sensor synchronization processing device according to one embodiment of the present invention.

[0087] Referring to FIG. 2, a sensor synchronization system (200) to which a sensor synchronization processing device according to one embodiment of the present invention is applied is illustrated.

[0088] A sensor synchronization system (200) according to one embodiment of the present invention is a synchronization system of a lidar sensor (220) and a camera (230) and can be introduced to various applications that use the corresponding sensors.

[0089] For example, the sensor synchronization system (200) synchronizes the lidar sensor (220) and the camera (230) through sensor synchronization processed through the sensor synchronization processing device (210).

[0090] The sensor synchronization processing device (210) can project a 3D point onto an image using the estimated camera internal and external parameters between the lidar sensor (220) and the camera sensor (230) using the following mathematical expression 1.

[0091] [Mathematical Formula 1]

[0092]

[0093] In mathematical expression 1, P i can represent a two-dimensional pixel projected onto the image plane, and P lcan represent a 3D point of point cloud data of a lidar sensor, R / t represents an external parameter including rotation and translation matrices that describe the coordinate transformation from the lidar coordinate system to the camera coordinate system, and C intrinsic can be supplemented with the following mathematical formula 2.

[0094] [Equation 2]

[0095]

[0096] C in Equation 2 intrinsic represents the internal matrix of the camera, and f x and f y represents the focal length in pixels, and c x and c y can represent a point close to the center of the image.

[0097] In mathematical expression 1, R / t can be supplemented by mathematical expression 3 below.

[0098] [Equation 3]

[0099]

[0100] Equation 3 represents a combination of rotation and translation, and the homogeneous transformation encoded with external parameters R and t represents the transformation from the LiDAR coordinate system l to the camera coordinate system c.

[0101] Therefore, P is the point P of LiDAR coordinates. l In the camera coordinate system, P is as follows: c is converted to .

[0102] The conversion of a 3D point in the lidar coordinate system into a 3D point in the camera coordinate system can be expressed as in mathematical equation 4.

[0103] [Equation 4]

[0104]

[0105] Mathematical expression 4 is the lidar coordinate X in mathematical expression 3. l , Y l and Zl Camera coordinate X reflecting c , Y c and Z c can be used to derive .

[0106] Using mathematical expressions 2 and 3, a 2D point can be obtained by projecting a 3D point of lidar coordinates onto an image based on mathematical expression 5.

[0107] [Equation 5]

[0108]

[0109] Sensor synchronization (200) estimates a delay offset between point cloud data and image data through a synchronization processing device (210), and can control the trigger out of the lidar sensor (220) and the trigger in of the camera sensor (230) for the delay offset.

[0110] FIG. 3 illustrates an autonomous driving system (300) to which a sensor synchronization processing device according to one embodiment of the present invention is applied.

[0111] Referring to FIG. 3, the autonomous driving system (300) is mounted on an autonomous driving vehicle and includes a lidar sensor (310) and a camera sensor (320).

[0112] The lidar sensor (310) and the camera sensor (320) can be configured as a sensor kit of the vehicle.

[0113] The vehicle's sensor kit is attached to the vehicle's rooftop and consists of three cameras and a 360-degree lidar sensor.

[0114] The internal clocks of all cameras and lidar sensors are synchronized with the clock of the sensor synchronization processing unit using PTP.

[0115] This clock synchronization allows all sensors to capture data with a common time reference.

[0116] The lidar sensor (310) can be placed at the top center of the sensor kit, and the camera sensors (320) can be placed at the center and on the left and right sides.

[0117] The autonomous driving system (300) can be utilized for the performance evaluation described in FIGS. 9a to 9d.

[0118] For example, the lidar sensor (310) operates at a frequency of 10 Hz, and the camera sensor (320) can operate at a frequency of up to 30 Hz depending on the approach method.

[0119] FIG. 4 is a drawing illustrating a lidar point projection of a sensor synchronization processing device for autonomous driving according to an embodiment of the present invention.

[0120] Referring to FIG. 4, image (400) may represent image data collected using a camera sensor, image (401) may represent point cloud data collected using a lidar sensor, and image (402) may represent fusion data of image data and point cloud data.

[0121] In data collection mode, the internal clocks of the lidar sensor and camera sensor can be synchronized using IEEE PTP.

[0122] Lidar and camera sensors can include time information when capturing data, facilitating frame alignment based on this time information during the fusion process.

[0123] t L can represent the timestamp of point cloud data collected using a lidar sensor.

[0124] t C can mean a timestamp on an image frame from a camera.

[0125] And the sensor data and timestamp can be expressed in vector form as in Equations 6 and 7.

[0126] [Equation 6]

[0127]

[0128] [Equation 7]

[0129]

[0130] L in Equation 6 S is the ith point cloud data and timestamp (t) contained in a single lidar scan. i L ) means.

[0131] C in Equation 7 f is the i-th image frame and timestamp (t i C ) can be expressed.

[0132] Unlike lidar sensors, which have a timestamp for each point cloud data, there is one timestamp for the entire image, not for each pixel.

[0133] After capturing data from the synchronized sensor, the image (401) can be projected onto the image (400) to generate an image (402).

[0134] [Equation 8]

[0135]

[0136] In mathematical expression 8, project refers to mathematical expression 5, which projects the lidar onto the image plane, and mathematical expression 8, which is an expression that maps the lidar point onto the image plane using a transformation matrix, can be expressed as mathematical expression 9.

[0137] [Equation 9]

[0138]

[0139] In equations 8 and 9, p d can represent the average projection error over n frames, and t d is the observed average delay time between the timestamp of the point cloud data from the lidar sensor and the timestamp of the image data from the camera sensor for n frames.

[0140] FIG. 5 is a drawing explaining reference point estimation of a lidar scan of a sensor synchronization processing device according to an embodiment of the present invention.

[0141] Referring to FIG. 5, an exemplary configuration (500) of a reference point estimation of a lidar scan of a sensor synchronization processing device according to an embodiment of the present invention is shown.

[0142] Example configuration (500) shows the reference point estimation results of the lidar scan for each connected camera sensor.

[0143] In the absence of a specific reference point, the average projection error (p d ) and average delay time (t d ) may not always be possible to estimate accurately.

[0144] These issues may arise from differences in data collection characteristics between lidar and camera sensors.

[0145] Lidar sensors scan a 360° area, while camera sensors can capture a limited field of view (FOV).

[0146] Additionally, the representative timestamps of the lidar sensor and the camera are different.

[0147] A timestamp from a camera may be when an image frame was captured, but a timestamp from a lidar sensor may be when the lidar sensor rotates and completes one scan.

[0148] As a result, it becomes difficult to find the point in time when the camera captures and the point in time when the LiDAR scans for the same FOV, and the average projection error (p d ) and average delay time (t d ) may affect the accuracy.

[0149] A sensor synchronization processing device according to one embodiment of the present invention can project point cloud data of a lidar sensor onto an image as in the example configuration (500) and select the point cloud projected closest to the center of the camera image as a reference point.

[0150] The vertical (φ) and horizontal (θ) LiDAR angles of this reference point are calculated as in the following mathematical expression 9.

[0151] [Equation 10]

[0152]

[0153] Additionally, the average projection error (p d ) and average delay time (t d ) is estimated to be closer to the center of the image.

[0154] This angle can be used as a reference point to initiate a trigger signal from the lidar sensor to the camera sensor.

[0155] FIG. 6 is a drawing illustrating a synchronization structure of a sensor synchronization processing device according to one embodiment of the present invention.

[0156] FIG. 6 illustrates a synchronization structure of a sensor synchronization processing device according to one embodiment of the present invention.

[0157] Referring to FIG. 6, the synchronization structure (600) of the sensor synchronization processing device according to one embodiment of the present invention can divide the lidar sensor (610) into a first lidar sensor state (611), a second lidar sensor state (612), and a third lidar sensor state (613) according to a reference time.

[0158] In addition, in the synchronization structure (600), the frame times of the first camera sensor (620), the second camera sensor (621), and the third camera sensor (622) are matched with the reference times of the first lidar sensor state (611), the second lidar sensor state (612), and the third lidar sensor state (613), thereby performing delay estimation and compensation.

[0159] The synchronization structure (600) can be performed through the synchronization algorithm described through FIG. 7 below.

[0160] FIG. 7 is a drawing illustrating a synchronization algorithm of a sensor synchronization processing device according to one embodiment of the present invention.

[0161] Referring to FIG. 7, a synchronization algorithm (700) of a sensor synchronization processing device according to one embodiment of the present invention is illustrated.

[0162] The synchronization algorithm (700) can be implemented through the synchronization structure described in FIG. 6.

[0163] In the synchronization algorithm (700), the initial trigger offset of the camera sensor is initially When the camera is set to , and a LiDAR trigger signal is received in the FOV matching the camera, the camera image is acquired.

[0164] Camera frame C in synchronization algorithm (700) f When receiving, the lidar sensor scans (L s ) is projected onto the image using a projection matrix.

[0165] Even if the lidar sensor can send a trigger signal directly to the camera, there may be an offset between the capture times of the camera and the lidar sensor.

[0166] This offset is caused by FOV mismatch between the camera and lidar sensor, internal camera processes, and signal transmission delays.

[0167] In the proposed method, the reference time of the lidar sensor and the reference time of the camera can be continuously matched through a series of processes of the algorithm, and the compensation parameters can be calculated in real time to adjust the camera shooting time.

[0168] When there are multiple cameras, the trigger signal from the lidar sensor generates a trigger signal for each camera at a different FOV, and the delay is calculated based on the reference point of the corresponding lidar sensor for each camera.

[0169] A trigger signal is generated and transmitted according to the camera FOV, as in the synchronization structure described in Fig. 6.

[0170] The compensation module can compensate for the estimated delay time by modulating the delay time of the camera.

[0171] FIG. 8 is a diagram illustrating projection error calculation related to synchronization performance of a sensor synchronization processing device according to an embodiment of the present invention.

[0172] Referring to Fig. 8, the result (800) of estimating the projection error of the lidar sensor and the camera sensor in a three-dimensional coordinate system is shown.

[0173] The time gap represents the time difference between the reference timestamp of the lidar sensor and the timestamp of the camera sensor.

[0174] Projection errors between data from lidar and camera sensors involve the process of estimating Euclidean distances between 3D points in world coordinates and transforming pixels in image coordinates into 3D coordinates.

[0175] To convert image pixels into world coordinates, the camera internal parameters and distance information provided by the lidar sensor can be utilized, as in the estimated result (800).

[0176] The internal parameters can be expressed by mathematical expressions 11 and 12.

[0177] [Equation 11]

[0178]

[0179] [Equation 12]

[0180]

[0181] Using mathematical expressions 11 and 12, the projection error can be derived as in mathematical expression 13.

[0182] [Equation 13]

[0183]

[0184] In mathematical expression 13, the point cloud data of the lidar sensor is is the camera coordinate system After being converted to , 3D projection error (p) is assigned to the corresponding pixel in the image as a Z value. d3D ) can be calculated.

[0185]

[0186] The accuracy of the projection alignment is measured by calculating the root mean square of the Euclidean distance between points on the lidar sensor and the camera sensor, and the lower the projection error, the more accurate the alignment between the data on the lidar sensor and the camera sensor.

[0187]

[0188] FIGS. 9A to 9D are drawings explaining the synchronization performance of a sensor synchronization processing device according to one embodiment of the present invention.

[0189] For each synchronization method in FIGS. 9a to 9d, the performance evaluation is conducted by dividing the scans into four times, 18,000 times based on LiDAR for a total of 30 minutes, and restarting the system and sensor for each scan to ensure fair evaluation.

[0190] To evaluate the performance of synchronization between the lidar sensor and the camera sensor, simulations can be performed by recording dynamic objects at various speeds and distances in several scenarios.

[0191] First, the experimental vehicle (hereinafter, experimental vehicle) on which the sensor system is installed is stopped, and the vehicle moving at various speeds at a certain distance is recorded to analyze the synchronization effect on the moving vehicle.

[0192] The test vehicle then drives at a constant speed and records other vehicles moving at various speeds and distances.

[0193] Additionally, we conduct a performance evaluation to verify the validity of the proposed method by comparing and analyzing various time synchronization approaches, including software synchronization, network synchronization, and hardware-level synchronization.

[0194] Referring to FIG. 9a, the results of fusion of frames of a lidar sensor and a camera sensor captured in software synchronization mode are exemplified in embodiments (900) and (901).

[0195] Example (900) illustrates a single static and dynamic object scenario, while Example (901) illustrates synchronization effects for multiple objects moving at different speeds and in different directions.

[0196] The fusion results according to Example (900) and Example (901) can be summarized as shown in Table 1 below.

[0197] [Table 1]

[0198]

[0199] According to Table 1, we can see that the mean and variance of the distance error increase linearly with the speed of the dynamic object, while the time difference remains the same.

[0200] Referring to FIG. 9b, the results of fusion of frames of a lidar sensor and a camera sensor captured in network synchronization mode are exemplified in embodiments (910) and (911).

[0201] Example (910) shows a single static and dynamic object scenario, and Example (911) shows the synchronization effect for multiple dynamic objects moving at different speeds and in different directions.

[0202] The fusion results according to Example (910) and Example (911) can be summarized as shown in Table 2 below.

[0203] [Table 2]

[0204]

[0205] According to Table 2, the network synchronization mode is asynchronous, making it difficult to match the acquisition time between sensors.

[0206] Increasing the camera's FPS will help reduce the time difference between data, but timing errors will still exist, and when using multiple cameras, bandwidth limitations will increase the likelihood of data transmission delays, which can further amplify synchronization errors.

[0207] Referring to FIG. 9c, the results of fusion of frames of a lidar sensor and a camera sensor captured in a hardware trigger synchronization mode are exemplified in embodiments (920) and (921).

[0208] Example (920) shows a single static and dynamic object scenario, and Example (921) shows the synchronization effect for multiple dynamic objects moving at different speeds and in different directions.

[0209] The fusion results according to Examples (920) and (921) can be summarized as shown in Table 3 below.

[0210] [Table 3]

[0211]

[0212] Referring to Table 3, the statistical analysis shows the estimated projection error and temporal difference error from frames captured over a 30-minute period.

[0213] The estimated mean and variance of projection errors increase continuously over time, especially with respect to the speed of moving objects.

[0214] Referring to FIG. 9d, the results of fusing frames of a lidar sensor and a camera sensor captured in a synchronization processing method according to an embodiment of the present invention are exemplified as examples (930) and (931).

[0215] Example (930) shows a single static and dynamic object scenario, and Example (931) shows the synchronization effect for multiple dynamic objects moving at different speeds and in different directions.

[0216] The fusion results according to Examples (930) and (931) can be summarized as shown in Table 4 below.

[0217] [Table 4]

[0218]

[0219]

[0220] Table 4 shows the results of an experimental analysis performed using a hardware trigger of the proposed delay compensation method with the goal of improving synchronization accuracy.

[0221] Delay compensation is calculated whenever the time gap between frames from the lidar sensor and the camera sensor exceeds a predefined threshold.

[0222] Then, based on the estimated compensation parameters, a camera trigger signal can be generated to initiate capture.

[0223] Quantitative comparison results in projection error show that the proposed hardware-based trigger method exhibits very small errors compared to the other three synchronization approaches.

[0224] Software synchronization-based methods were found to be prone to projection errors because they match frames between the LiDAR sensor and the camera sensor using ROS software callback functions.

[0225] For network synchronization, timestamp-based frame matching seems theoretically promising, but may result in poor performance in real-time applications.

[0226] Especially when the system includes multiple sensors, performance degrades due to network overload and transmission delays.

[0227] Hardware trigger-based synchronization appears to be a potential solution to these synchronization problems.

[0228] However, errors may accumulate during projection due to variations in the trigger signal and delays in data transmission from the sensor.

[0229] Finally, the proposed synchronization method significantly improves synchronization accuracy by reducing the projection error to less than 10 cm through real-time monitoring and delay compensation.

[0230] The present invention proposes an automatic delay compensation to alleviate dynamic delay problems caused by various factors in order to address the limitations of hardware trigger-based synchronization approaches.

[0231] In the proposed method, the time difference between frames of the lidar sensor and the camera sensor observed at regular intervals and the compensation parameters are precisely calculated and synchronization is adjusted accordingly.

[0232] FIG. 10 is a drawing explaining a sensor synchronization processing method according to one embodiment of the present invention.

[0233] FIG. 10 illustrates a procedure for performing synchronization of a lidar sensor and a camera sensor in a sensor synchronization processing method according to one embodiment of the present invention.

[0234] Referring to FIG. 10, in step (S1001), a sensor synchronization processing method according to an embodiment of the present invention collects point cloud data and image data.

[0235] That is, a sensor synchronization processing method according to one embodiment of the present invention can collect point cloud data and image data using a lidar sensor and multiple camera sensors.

[0236] In step (S1002), a sensor synchronization processing method according to one embodiment of the present invention estimates a delay offset between point cloud data and image data.

[0237] That is, a sensor synchronization processing method according to one embodiment of the present invention can select a reference point by projecting point cloud data onto image data, and estimate a delay offset based on the selected reference point.

[0238] In step (S1003), a sensor synchronization processing method according to one embodiment of the present invention performs delay compensation to match the reference times of the lidar sensor and multiple camera sensors.

[0239] That is, a sensor synchronization processing method according to an embodiment of the present invention can perform delay compensation by correcting the shooting time of each of the plurality of camera sensors by calculating a compensation parameter that matches the reference time of the lidar sensor and the reference time of each of the plurality of camera sensors based on the estimated delay offset.

[0240] In step (S1004), a sensor synchronization processing method according to an embodiment of the present invention performs synchronization of a lidar sensor and a plurality of camera sensors.

[0241] That is, the sensor synchronization processing method according to one embodiment of the present invention can perform synchronization of the lidar sensor and the plurality of camera sensors based on the performed delay compensation.

[0242] In other words, a sensor synchronization processing method according to an embodiment of the present invention can perform synchronization of a lidar sensor and a plurality of camera sensors based on a delay time of a plurality of camera sensors determined according to a compensation parameter for a reference time difference between heterogeneous sensors that fluctuates due to various reasons (at least one of heat generation, communication delay, and computational delay).

[0243] Therefore, the present invention provides a synchronization function of a lidar sensor and multiple camera sensors, thereby overcoming complex delay time problems depending on the system environment, network load, and sensor status in real time.

[0244] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0245] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0246] Although the embodiments described above have been described with limited drawings, those skilled in the art will recognize that various modifications and variations can be made based on the above description. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0247] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. A data collection unit that collects point cloud data and image data using a LiDAR sensor and multiple camera sensors; A delay offset estimation unit that projects the point cloud data onto the image data to select a reference point and estimates a delay offset based on the selected reference point; A delay compensation unit that performs delay compensation by correcting the shooting time of each of the plurality of camera sensors by calculating a compensation parameter that matches the reference time of the lidar sensor and the reference time of each of the plurality of camera sensors based on the estimated delay offset; and It is characterized by including a synchronization processing unit that performs synchronization of the lidar sensor and the plurality of camera sensors based on the delay compensation performed above. Sensor synchronization processing unit.

2. In paragraph 1, The delay offset estimation unit is characterized in that it converts the time stamp of the point cloud data and the time stamp of the image data into a vector, and estimates the average projection error and the average delay time by projecting and mapping the 3D lidar point of the point cloud data onto the image plane using a transformation matrix. Sensor synchronization processing unit.

3. In paragraph 2, The delay offset estimation unit is characterized in that it projects and maps the 3D lidar point of the point cloud data onto the image plane and selects a reference point based on the distance between the center of the camera image and the lidar point projected onto the image. Sensor synchronization processing unit.

4. In paragraph 3, The delay offset estimation unit is characterized in that it calculates the vertical and horizontal lidar angles for the selected reference point and determines the angles at which each of the plurality of camera sensors outputs a trigger signal. Sensor synchronization processing unit.

5. In paragraph 1, The delay compensation unit is characterized in that it performs delay compensation to compensate for the estimated delay offset by correcting the shooting time of each of the plurality of camera sensors using the delay time of each of the plurality of camera sensors for the lidar sensor. Sensor synchronization processing unit.

6. In paragraph 1, The synchronization processing unit is characterized in that it performs synchronization of the lidar sensor and the plurality of camera sensors based on the trigger signal of the plurality of camera sensors determined according to the compensation parameter to compensate for the reference time difference between the heterogeneous sensors that fluctuates due to at least one of heat generation, communication delay, and operation delay. Sensor synchronization processing unit.

7. In the data collection unit, a step of collecting point cloud data and image data using a LiDAR sensor and multiple camera sensors; In the delay offset estimation unit, a step of projecting the point cloud data onto the image data to select a reference point, and estimating a delay offset based on the selected reference point; In the delay compensation unit, a step of performing delay compensation by calculating a compensation parameter that matches the reference time of the lidar sensor with the reference time of each of the plurality of camera sensors based on the estimated delay offset and correcting the shooting time of each of the plurality of camera sensors; and In the synchronization processing unit, a step of performing synchronization of the lidar sensor and the plurality of camera sensors based on the performed delay compensation is included. How to handle sensor synchronization.

8. In paragraph 7, The step of collecting point cloud data and image data using the above LiDAR sensor and multiple camera sensors is as follows: A method for obtaining a 2D point by projecting a 3D lidar point of the point cloud data onto a 2D pixel of the image data using internal and external parameters for the plurality of camera sensors, thereby converting coordinates from the lidar coordinate system to the camera coordinate system. How to handle sensor synchronization.

9. In paragraph 7, The step of projecting the above point cloud data onto the above image data to select a reference point and estimating a delay offset based on the selected reference point is as follows. A method characterized by comprising the steps of converting a time stamp of the point cloud data and a time stamp of the image data into vectors, and projecting and mapping a 3D lidar point of the point cloud data onto an image plane using a transformation matrix to estimate an average projection error and an average delay time. How to handle sensor synchronization.

10. In paragraph 9, The step of projecting the above point cloud data onto the above image data to select a reference point and estimating a delay offset based on the selected reference point is as follows. A step of projecting and mapping the 3D lidar points of the above point cloud data onto an image plane and selecting a reference point based on the distance between the center of the camera image and the projected lidar points of the image; and A step of calculating vertical and horizontal lidar angles for the selected reference point and determining the selected reference point as an angle at which each of the plurality of camera sensors starts a trigger signal for the lidar sensor. How to handle sensor synchronization.

11. In paragraph 7, The step of performing delay compensation by correcting the shooting time of each of the plurality of camera sensors is as follows: A step of performing delay compensation to compensate for the estimated delay offset by correcting the shooting time of each of the plurality of camera sensors using the delay time of each of the plurality of camera sensors for the lidar sensor. How to handle sensor synchronization.

12. In paragraph 7, The step of performing synchronization of the lidar sensor and the plurality of camera sensors based on the delay compensation performed above is: A step of synchronizing the lidar sensor and the plurality of camera sensors based on trigger signals of the plurality of camera sensors determined according to the compensation parameter, wherein the reference time difference between the heterogeneous sensors fluctuates due to at least one of heat generation, communication delay, and computational delay. How to handle sensor synchronization.

Images