Time synchronization system and method between camera sensor and lidar sensor

Abstract

This invention provides a time synchronization system and method between a camera sensor and a lidar sensor that can improve the reliability of the data. [Solution] A first detection unit analyzes image data to detect moving object data, a second detection unit analyzes point cloud data to group the detected moving object data, and calculates the center point position of each moving object data. Using the velocity and acceleration of each moving object, the camera sensor and lidar sensor are synchronized in time to improve calibration errors caused by differences in data collection times.

Description

Technical Field

[0001] The present invention relates to a time synchronization system and method for a camera sensor and a lidar sensor, and more particularly, to a time synchronization system and method for a camera sensor and a lidar sensor that can synchronize the data of the camera sensor and the lidar sensor in a software-based and real-time manner by using the movement of a moving object in the acquired data.

Background Art

[0002] In a traffic control system, a camera sensor and a lidar sensor, which are mainly used for object recognition and judgment, are generally used in combination to complement the vulnerabilities of each sensor.

[0003] The camera sensor provides various shape information such as texture and hue, but has the disadvantage of being difficult to measure distance. The lidar sensor can accurately measure distance, but has the disadvantage of being difficult to obtain precise shape information.

[0004] For such fusion of a camera sensor and a lidar sensor, accurate calibration (the process of matching the camera coordinate system and the lidar coordinate system) is essential, and it has mainly been performed in an offline manner using a target such as a checkerboard.

[0005] However, in actual use, even if accurate calibration parameters are obtained, calibration errors accumulate due to time synchronization problems between the camera sensor and lidar sensor that occur during use. When such errors occur, the reliability of sensor fusion information decreases, resulting in the disadvantage of not being able to use it effectively. In other words, even if image data and point cloud data are collected at predetermined intervals at specific pre-set time points, minute time differences may occur due to various external / internal conditions. In this case, as illustrated in Figure 1, errors in shape information occur due to the point cloud data matching the shape information obtained by analyzing the image data.

[0006] Traditionally, time synchronization between camera and LiDAR sensors was achieved using hardware to mount them and signal triggers generated by data acquisition. However, this required the sensors to be physically close together, which has been a disadvantage in traffic control systems requiring a wide field of view.

[0007] Therefore, the time synchronization system and method for camera sensors and lidar sensors of the present invention provides a technology that synchronizes data from camera sensors and lidar sensors in a software-based and real-time manner using the movement of a moving object, thereby freeing oneself from the physical constraints of conventional time synchronization that have been performed using hardware and signal triggers from data acquisition.

[0008] In this regard, Korean Published Patent No. 10-2024-0072677 ("Method and Apparatus for Synchronizing Phase and Time of LiDAR-Camera") discloses a technology that can synchronize the phase and time of a LiDAR with a number of cameras. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] Korean Published Patent No. 10-2024-0072677 (Published on May 24, 2024) [Overview of the Initiative] [Problems that the invention aims to solve]

[0010] Therefore, the present invention was derived to solve the problems of the prior art described above, and aims to provide a time synchronization system and method for a camera sensor and a lidar sensor that can improve calibration errors caused by differences in data acquisition times and improve data reliability by using the velocity and acceleration information of moving objects in the acquired data to synchronize the camera sensor and the lidar sensor. [Means for solving the problem]

[0011] A time synchronization system between a camera sensor and a lidar sensor according to one embodiment of the present invention for achieving the above-mentioned objectives includes: a first detection unit that analyzes image data input via a pre-stored first data processing algorithm to detect moving object data and cluster the detected moving object data; a second detection unit that analyzes point cloud data input via a pre-stored second data processing algorithm to detect moving object data and cluster the detected moving object data; a first estimation unit that calculates the center point position of each moving object data clustered by the first detection unit and uses this to estimate motion-related information of each moving object data; and an image processing unit that processes each moving object data clustered by the second detection unit. Preferably, the system includes: a second estimation unit that, after projecting to the Xel coordinate system and performing a coordinate system transformation, calculates the center point position based on the position coordinates in the transformed image pixel coordinate system and uses this to estimate motion-related information for each moving object data; an object determination unit that, based on the image pixel coordinate system, analyzes the center point position of each moving object data determined by the first detection unit and the center point position based on the position coordinates in the transformed image pixel coordinate system obtained by the second estimation unit to determine that they are the same moving object data; and a synchronization processing unit that compares the motion-related information of each moving object data determined to be the same moving object data, calculates a time offset, and uses the calculated time offset to perform time synchronization between the camera sensor and the lidar sensor.

[0012] Furthermore, it is preferable that the motion-related information includes the velocity and acceleration information of the moving object.

[0013] Furthermore, it is preferable that the synchronization processing unit compares the velocity information of each moving object data that has been determined to be the same moving object data, and calculates the time offset of the camera sensor and lidar sensor corresponding to each image data and point cloud data.

[0014] Furthermore, it is preferable that the synchronization processing unit applies the calculated time offset to the velocity information of the moving object data from the point cloud data, corrects the translation vector of each moving object data, and performs time synchronization between the camera sensor and the lidar sensor.

[0015] Another embodiment of the present invention provides a time synchronization method between a camera sensor and a lidar sensor, comprising a time synchronization system for a camera sensor and a lidar sensor in which each step is performed by a calculation processing means, comprising: a first detection step (S100) in which a first detection unit analyzes image data from the input camera sensor via a pre-stored first data processing algorithm to detect moving object data and clusters the detected moving object data; a second detection step (S200) in which a second detection unit analyzes point cloud data from the input lidar sensor via a pre-stored second data processing algorithm to detect moving object data and clusters the detected moving object data; a first estimation step (S300) in which a first estimation unit calculates the center point position of each moving object data clustered in the first detection step (S100) and uses this to estimate motion-related information for each moving object data; and a second estimation unit processes each moving object data clustered in the second detection step (S200) A coordinate transformation step (S400) is performed by projecting onto the image pixel coordinate system and performing a coordinate system transformation; a second estimation step (S500) is performed in the second estimation unit, where the center point position is calculated based on the position coordinates in the image pixel coordinate system transformed by the coordinate transformation step (S400), and motion-related information for each moving object data is estimated using this; and an object determination unit uses the image pixel coordinate system as a reference to the center point position for each moving object data calculated in the first estimation step (S300) and the first detection step (S100), and the second The object determination step (S600) analyzes the center point position for each moving object data calculated in the estimation step (S400) by the second detection step (S200) and determines that they are the same moving object data. The synchronization processing unit compares the motion-related information of each moving object data determined to be the same moving object data in the object determination step (S600) and calculates an offset calculation step (S700). The synchronization processing unit uses the time offset calculated in the offset calculation step (S700) to perform the following:It is preferable to include a synchronization processing step (S800) that synchronizes the time between the camera sensor and the lidar sensor.

[0016] Furthermore, the object determination step (S600) preferably analyzes the center point position for each moving object data calculated in the first detection step (S100) in the first estimation step (S300) and the center point position for each moving object data calculated in the second detection step (S200) in the second estimation step (S400), based on the image pixel coordinate system, and determines that the pair of moving object data corresponding to the most adjacent center points are the same moving object data.

[0017] Furthermore, it is preferable that the motion-related information obtained in the first estimation step (S300) and the second estimation step (S500) includes velocity and acceleration information of the moving object.

[0018] Furthermore, it is preferable that the offset calculation step (S700) compares the velocity information of each moving object data determined to be the same moving object data in the object determination step (S600) and calculates the time offset of the camera sensor and lidar sensor corresponding to each image data and point cloud data.

[0019] Furthermore, it is preferable that the synchronization processing step (S800) applies the time offset calculated in the offset calculation step (S700) to the velocity information of the moving object data from the point cloud data, corrects the translation vector of each moving object data, and performs time synchronization between the camera sensor and the lidar sensor. [Effects of the Invention]

[0020] According to the present invention, the time synchronization system and method for a camera sensor and a lidar sensor have the advantage of correcting calibration errors between the two sensors (camera sensor and lidar sensor) caused by camera shake via a motionless background image, thereby ensuring the stability of the image data.

[0021] That is, by using the speed and acceleration of a moving object, and improving the calibration error caused by the difference in data collection time through time synchronization between the camera sensor and the lidar sensor, there is an advantage that the reliability of the data can be improved.

Brief Description of Drawings

[0022] [Figure 1] It is an exemplary diagram showing the calibration error due to the time difference between the camera sensor and the lidar sensor. [Figure 2] It is an exemplary configuration diagram showing the time synchronization system between a camera sensor and a lidar sensor according to an embodiment of the present invention. [Figure 3] In the time synchronization system and method between a camera sensor and a lidar sensor according to an embodiment of the present invention, it is an exemplary diagram showing the separation of moving object data and background object data by analyzing image data. [Figure 4] In the time synchronization system and method between a camera sensor and a lidar sensor according to an embodiment of the present invention, it is an exemplary diagram showing the separation of moving object data and static data by analyzing point cloud data. [Figure 5] In the time synchronization system and method between a camera sensor and a lidar sensor according to an embodiment of the present invention, it is an exemplary diagram showing the result of clustering the moving object data detected by analyzing point cloud data. [Figure 6] It is an exemplary sequence diagram showing the time synchronization method between a camera sensor and a lidar sensor according to an embodiment of the present invention.

Modes for Carrying Out the Invention

[0023] The time synchronization system and method for a camera sensor and a lidar sensor according to the present invention, having the configuration described above, will be described in detail below with reference to the accompanying drawings. The drawings presented below are provided as examples so that the concept of the present invention may be fully conveyed to those skilled in the art. Therefore, the present invention is not limited to the drawings presented below and may be embodied in other forms. Also, the same reference numerals throughout the specification indicate the same components.

[0024] Herein, unless otherwise defined, technical and scientific terms have the meanings that a person with ordinary skill in the art to which this invention belongs, and in the following description and accompanying drawings, descriptions of known functions and configurations that could obscure the gist of this invention are omitted.

[0025] Furthermore, a system refers to a collection of components, including devices, mechanisms, and means, that are organized and interact regularly in order to perform a required function.

[0026] A time synchronization system and method for a camera sensor and a lidar sensor according to one embodiment of the present invention assumes that calibration has been performed in advance using an offline method, using a target such as a checkerboard, and the camera internal parameter K and external parameter acquired by the offline calibration

number

[0027] Here, the internal parameters include the distance between the lens and the image sensor, and the angle between the lens and the image sensor, when projecting the three-dimensional space of the real world onto a two-dimensional image plane. The external parameters consist of the camera's position and orientation in the three-dimensional scene (the three-dimensional scene of the LiDAR sensor and the camera sensor), rotation, and translation.

[0028] Figure 2 is an illustrative diagram showing a time synchronization system between a camera sensor and a lidar sensor according to one embodiment of the present invention. As shown in Figure 2, the time synchronization system between a camera sensor and a lidar sensor according to one embodiment of the present invention includes a first detection unit 100, a second detection unit 200, a first estimation unit 300, a second estimation unit 400, an object determination unit 500, and a synchronization processing unit 600. Each component operates using arithmetic processing means including a CPU.

[0029] To explain each step in detail: Preferably, the first detection unit 100 analyzes the input image data via a pre-stored first data processing algorithm to detect moving object data and then aggregates the detected moving object data.

[0030] The aforementioned image data refers to data collected from the camera sensor, and represents a set of x and y coordinates at any given time and on the image plane, as shown in Equation 2 below.

[0031]

number

[0032] The first detection unit 100 analyzes the input image data I using a pre-stored first data processing algorithm, which is a background removal algorithm. Here, the pre-stored first data processing algorithm is not necessarily limited to a background removal algorithm; any data processing algorithm capable of separating moving objects from the background in image data may be used.

[0033] As a result of the analysis by the first detection unit 100, as shown in Figure 3, moving object data I moving (Defined as a set of 2D pixel values ​​(x, y) in 2D image data measured by a camera sensor, where the pixel position and velocity values ​​change over time.) and background data I bg(Defined as a set of 2D pixel values ​​in 2D image data measured by a camera sensor, where the pixel position and velocity values ​​do not change over time.) This can be separated into these, and can be defined as shown in equation 3 below.

[0034]

number

[0035] Next, the first detection unit 100 performs crowding on the pixel sets included in the detected moving object data and generates separate moving object data for each.

[0036] Herein, for the sake of smooth explanation, the present invention will be described in part in a limited manner when applied to a traffic control system that collects and analyzes data while monitoring traffic conditions in real time, prevents traffic accidents, and optimizes traffic flow.

[0037] Accordingly, the data of each moving object generated according to the crowding results performed by the first detection unit 100 may be interpreted as data of each moving vehicle. The application to the traffic control system is merely one embodiment of the present invention and is applicable to various technological fields that utilize camera sensors and lidar sensors simultaneously.

[0038] Preferably, the second detection unit 200 analyzes the point cloud data (point cloud data) input via a pre-stored second data processing algorithm to detect moving object data and then aggregates the detected moving object data.

[0039] The aforementioned point cloud data represents light reflected from an object as points, as unstructured data measured from a lidar sensor. This is expressed as a set of three-dimensional points by measuring x, y, z, and intensity (unstructured data), as shown in Equation 4 below.

[0040]

number

[0041] The second detection unit 200 analyzes the input point cloud data using the Max-distance algorithm, which is a pre-stored second data processing algorithm. Here, the pre-stored second data processing algorithm is not necessarily limited to the Max-distance algorithm; any data processing algorithm capable of distinguishing between static and dynamic data in point cloud data may be used.

[0042] As shown in Figure 4, the analysis results by the second detection unit 200 can be separated into static point cloud data and dynamic point cloud data, which can be defined as shown in equation 5 below. Here, the separated dynamic point cloud data is limited to moving object data and detected.

[0043]

number

[0044] Next, the second detection unit 200 uses the DBSCAN crowding method, a pre-stored data processing algorithm, to perform crowding on the detected moving object data, which is dynamic point cloud data, as shown in Equation 6 below. This removes unnecessary noise (such as ground point data) from the moving object data, as illustrated in Figure 5, and improves computational efficiency.

[0045]

number

[0046] Preferably, the first estimation unit 300 calculates the center point position for each moving object data using the image data aggregated by the first detection unit 100, and uses this to estimate motion-related information for each moving object data.

[0047] In detail, the first estimation unit 300 uses the image data aggregated by the first detection unit 100 to obtain data for each moving object.

number

[0048]

number

[0049] Using the center point position calculated in this way, the velocity and acceleration information of each moving object data, which are motion-related information for each moving object data, are used.

number

[0050]

number

[0051] Furthermore, the first estimation unit 300 preferably applies a Kalman filter to reduce noise in the center point position and velocity data of the moving object and to estimate the motion-related information more precisely. The state vector for this purpose is defined as shown in Equation 11 below.

[0052]

number

[0053] The first estimation unit 300 uses a positive matrix A in which both the camera sensor and the lidar sensor are in the same state, which is defined as shown in equation 12 below.

[0054]

number

[0055] Furthermore, the observation matrix H, which considers only position observations, is now applied similarly to the camera sensor and the lidar sensor, and this is defined as shown in equation 13 below.

[0056]

number

[0057] The second estimation unit 400 projects each moving object data from the point cloud data aggregated by the second detection unit 200 onto the image pixel coordinate system and performs a coordinate system transformation. Next, it is preferable to calculate the center point position based on the position coordinates in the transformed image pixel coordinate system and use this to estimate the motion-related information of each moving object data.

[0058] In detail, as described above, the camera's internal parameter K and external parameter are calibrated using a pre-performed offline method.

number

[0059] Based on these points, the second estimation unit 400 projects each moving object data from the point cloud data aggregated by the second detection unit 200 onto the image by projection transformation, and the position in the image pixel coordinate system is determined as shown in Equation 16 below.

number

[0060]

number

[0061] The second estimation unit 400 uses the position coordinates in the transformed image pixel coordinate system as a reference for each moving object data

number

[0062]

number

[0063] Using the center point position calculated in this way, the velocity and acceleration information of each moving object data, which are motion-related information for each moving object data, are used.

number

[0064]

number

[0065] Furthermore, it is preferable that the second estimation unit 400 also applies a Kalman filter to reduce noise in the data, in other words, to reduce noise in the center point position and velocity data of the moving object, similar to the first estimation unit 300, and to estimate the motion-related information more precisely.

[0066] It is preferable that the object determination unit 500 analyzes the center point position based on the image pixel coordinate system, the center point position of each moving object data obtained by the first detection unit 100, and the position coordinates in the image pixel coordinate system converted by the second estimation unit 400, and determines that the data is the same moving object.

[0067] In detail, the object determination unit 500 preferably detects moving objects from image data from the camera sensor via the first detection unit 100 and the first estimation unit 300, and then detects moving objects from point cloud data from the lidar sensor via the second detection unit 200 and the second estimation unit 400. After that, it preferably applies a Hungarian algorithm based on the center point position of the moving objects on the image pixel coordinate system to form a pair of moving objects whose center point positions are closest to each other and determines that they are the same object.

[0068] In other words, by comparing the deceleration and acceleration states of the moving object data detected by the camera sensor and the lidar sensor at a specific point in time, it is possible to calculate the time offset between the two sensors and determine which sensor's data is superior.

[0069] However, this requires comparing the speed of objects using the same moving object data (pairs of moving objects) detected by the camera sensor and the lidar sensor. Therefore, the object determination unit 500 applies a Hungarian algorithm based on the center point position of each moving object data from each sensor, and treats the moving object data from the closest adjacent, different sensors as the same moving object data, i.e., as a pair of moving objects.

[0070] Preferably, the synchronization processing unit 600 compares the motion-related information of each moving object data that the object determination unit 500 has determined to be the same moving object data, calculates a time offset, and uses the calculated time offset to perform time synchronization between the camera sensor and the lidar sensor.

[0071] In detail, at the point when data from each sensor (image sensor and lidar sensor) is input.

number

[0072]

number

[0073] The synchronization processing unit 600 compares the motion-related information of each motion-related data, which is determined to be the same motion-related data, in other words, the motion-related data from the motion-related data obtained from the image data and the motion-related data obtained from the point cloud data, which is determined to be the same motion-related data. If the acceleration of the motion-related data is positive at a particular point in time, and the velocity of the motion-related data obtained from the camera sensor is estimated to be faster than the velocity of the motion-related data obtained from the LiDAR sensor, the synchronization processing unit 600 analyzes this as the image data obtained from the camera sensor being input later.

[0074] Conversely, if the acceleration of the moving object is a negative number, and the velocity of the moving object estimated by the camera sensor is faster than the velocity of the moving object estimated by the lidar sensor, this is analyzed as the point cloud data from the lidar sensor being input later.

[0075] Using these points, the time offset was calculated.

number

[0076]

number

[0077] To explain the correction process in detail, the 3D velocity information of the moving object data estimated using the LiDAR sensor is used.

number

number

[0078] The moving object data from the LiDAR sensor is obtained through an offline calibration performed in advance, as shown in the number 28 below, using external parameters.

number

[0079]

number

[0080] Next, the synchronization processing unit 600 calculates 3D velocity information using the change in position of the moving object's center point in the camera coordinate system, as shown in equation 29 below.

[0081]

number

[0082] Next, the synchronization processing unit 600 calculates the 3D velocity information and time offset.

number

number

[0083]

number

[0084] The synchronization processing unit 600 processes the corrected translation vector

number

[0085]

number

[0086] Figure 6 is a sequence diagram illustrating a time synchronization method between a camera sensor and a lidar sensor according to one embodiment of the present invention. As shown in Figure 6, the time synchronization method between a camera sensor and a lidar sensor according to one embodiment of the present invention includes a first detection step (S100), a second detection step (S200), a first estimation step (S300), a coordinate transformation step (S400), a second estimation step (S500), an object determination step (S600), an offset calculation step (S700), and a synchronization processing step (S800). Each step is performed by a time synchronization system between a camera sensor and a lidar sensor, in which each step is performed by a calculation processing means.

[0087] To explain each step in detail: The first detection step (S100) involves the first detection unit 100 analyzing the input image data via a pre-stored first data processing algorithm to detect moving object data and then grouping the detected moving object data.

[0088] The aforementioned image data refers to data collected from the camera sensor and, as shown in equation 2 above, represents a set of x and y coordinates at any given time and on the image plane.

[0089] The first detection step (S100) analyzes the input image data I using a pre-stored first data processing algorithm, which is a background removal algorithm. Here, the pre-stored first data processing algorithm is not necessarily limited to a background removal algorithm; any data processing algorithm capable of separating the moving object from the background in the image data may be used.

[0090] As a result of the analysis performed in the first detection step (S100), as shown in Figure 3, the moving object data I moving (Defined as a set of 2D pixel values ​​(x, y) in 2D image data measured by a camera sensor, where the pixel position and velocity values ​​change over time.) and background data I bg (Defined as a set of 2D pixel values ​​in 2D image data measured by a camera sensor, where the pixel position and velocity values ​​do not change over time.) This can be separated into these, which can be defined as shown in equation 3 above.

[0091] Next, the first detection step (S100) performs crowding on the pixel sets included in the detected moving object data and generates separate moving object data for each.

[0092] Herein, for the sake of smooth explanation, the present invention will be described in part in a limited manner when applied to a traffic control system that collects and analyzes data while monitoring traffic conditions in real time, prevents traffic accidents, and optimizes traffic flow.

[0093] Therefore, each moving object data generated according to the crowding result of the first detection step (S100) may be interpreted as each moving vehicle data. The application to the traffic control system is merely one embodiment of the present invention, and it goes without saying that it is applicable to various technological fields that utilize camera sensors and lidar sensors simultaneously.

[0094] The second detection step (S200) involves the second detection unit 200 analyzing point cloud data (point cloud data) input via a pre-stored second data processing algorithm to detect moving object data and then grouping the detected moving object data. The point cloud data represents light reflected from objects as points, as unstructured data measured from a lidar sensor. This is represented by measuring and expressing x, y, z, and intensity (unstructured data) as a set of three-dimensional points, as shown in equation 4 above.

[0095] The second detection step (S200) analyzes the input point cloud data using the Max-distance algorithm, which is a pre-stored second data processing algorithm. Here, the pre-stored second data processing algorithm is not necessarily limited to the Max-distance algorithm; any data processing algorithm capable of distinguishing between static and dynamic data in point cloud data may be used.

[0096] As a result of the analysis performed in the second detection step (S200), as illustrated in Figure 4, static point cloud data and dynamic point cloud data can be separated, which can be defined as shown in equation 5 above. Here, the separated dynamic point cloud data is limited to detecting moving object data.

[0097] Next, the second detection step (S200) uses the DBSCAN crowding method, a pre-stored data processing algorithm, to perform crowding on the detected moving object data, which is dynamic point cloud data, as shown in equation 6 above. This removes unnecessary noise (such as ground point data) from the moving object data, as illustrated in Figure 5, and improves computational efficiency.

[0098] In the first estimation step (S300), the first estimation unit 300 calculates the center point position for each moving object data using the image data aggregated in the first detection step (S100), and uses this to estimate motion-related information for each moving object data.

[0099] The first estimation step (S300) involves collecting data for each moving object from the image data aggregated by the first detection step (S100).

number

[0100] The first estimation step (S300) uses the calculated center point position to obtain motion-related information for each moving object data, which is the velocity and acceleration information for each moving object data.

number

[0101] In the first estimation step (S300), it is preferable to apply a Kalman filter to reduce noise in the center point position and velocity data of the moving object and to estimate the motion-related information more precisely. The state vector for this purpose is defined as shown in equation 11 above.

[0102] The first estimation step (S300) uses a positive matrix A where both the camera sensor and the lidar sensor are in the same state, which is defined as shown in number 12 above.

[0103] Furthermore, the observation matrix H, which considers only position observations, is applied similarly to the camera sensor and the lidar sensor, and this is defined as shown in equation 13 above.

[0104] The coordinate transformation step (S400) involves the second estimation unit 400 projecting the data of each moving object, which has been grouped by the second detection step (S200), onto the image pixel coordinate system to perform a coordinate system transformation.

[0105] In detail, as described above, the camera's internal parameter K and external parameter are calibrated using a pre-performed offline method.

number

[0106] Based on these points, the coordinate transformation step (S400) projects each moving object data from the point cloud data aggregated by the second detection step (S200) onto the image by projection transformation, and the position in the image pixel coordinate system is determined as shown in equation 16 above.

number

[0107] In the second estimation step (S500), the second estimation unit 400 calculates the center point position based on the position coordinates in the image pixel coordinate system transformed by the coordinate transformation step (S400), and uses this to estimate motion-related information for each moving object data.

[0108] In detail, the second estimation step (S500) uses the position coordinates in the image pixel coordinate system transformed by the coordinate transformation step (S400) as a reference for each moving object data

number

[0109] Using the center point position calculated in this way, the velocity and acceleration information of each moving object data, which are motion-related information for each moving object data, are used.

number

[0110] In the second estimation step (S500), similar to the first estimation step (S300), it is preferable to apply a Kalman filter to reduce noise in the data, thereby reducing noise in the center point position and velocity data of the moving object and estimating the motion-related information with greater precision.

[0111] In the object determination step (S600), the object determination unit 500 analyzes the center point position for each moving object data calculated in the first detection step (S100) in the first estimation step (S300) and the center point position for each moving object data calculated in the second detection step (S200) in the second estimation step (S400), based on the image pixel coordinate system, and determines that they are the same moving object data.

[0112] In detail, the object determination step (S600) applies the Hungarian algorithm based on the center point positions for each moving object data calculated in the first estimation step (S300) and the center point positions for each moving object data calculated in the second estimation step (S400) and the second detection step (S200), to form a pair of moving objects whose center point positions are closest to each other, and determines that they are the same object.

[0113] In other words, by comparing the speed of the moving object based on the deceleration and acceleration states of the data from the camera sensor and the lidar sensor detected at a specific point in time, it is possible to calculate the time offset between the two sensors and determine which sensor's data is superior.

[0114] However, this requires comparing the speed of objects using the same moving object data (pairs of moving objects) detected by the camera sensor and the lidar sensor. Therefore, the object determination step (S600) applies the Hungarian algorithm based on the center point position of each moving object data from each sensor, and treats the moving object data from the closest adjacent, different sensors as the same moving object data, that is, as a pair of moving objects.

[0115] The offset calculation step (S700) involves the synchronization processing unit 600 comparing the motion-related information of each moving object data that has been determined to be the same moving object data in the object determination step (S600) to calculate a time offset.

[0116] In detail, the offset calculation step (S700) is performed when data from each sensor (image sensor and lidar sensor) is input.

number

[0117] The offset calculation step (S700) compares the motion-related information of each moving object data that has been determined to be the same moving object data, in other words, the moving object data from image data that has been determined to be the same moving object data and the moving object data from point cloud data that has been determined to be the same moving object data. If the acceleration of the moving object is a positive number at a particular point in time and the velocity of the moving object estimated by the camera sensor is faster than the velocity of the moving object estimated by the lidar sensor, this is analyzed as the image data from the camera sensor being input later.

[0118] Conversely, if the acceleration of the moving object is a negative number, and the velocity of the moving object estimated by the camera sensor is faster than the velocity of the moving object estimated by the lidar sensor, this is analyzed as the point cloud data from the lidar sensor being input later.

[0119] Using these points, the time offset was calculated.

number

[0120] The synchronization processing step (S800) involves the synchronization processing unit 600 performing time synchronization between the camera sensor and the lidar sensor using the time offset calculated in the offset calculation step (S700).

[0121] In detail, the synchronization processing step (S800) involves 3D velocity information of moving object data estimated using a lidar sensor.

number

number

[0122] The moving object data from the LiDAR sensor is obtained through an external parameter acquired by a pre-performed offline calibration, as shown in number 28 above.

number

[0123] Next, the synchronization processing step (S800) calculates 3D velocity information using the change in position of the center point of the moving object in the camera coordinate system, as shown in equation 29 above.

[0124] Next, the synchronization processing step (S800) calculates the 3D velocity information and time offset

number

number

[0125] Next, the synchronization processing step (S800) corrects the translation vector

number

[0126] In other words, the time synchronization system and method for camera sensors and lidar sensors according to one embodiment of the present invention are applicable to fields such as traffic control systems, autonomous vehicles, and security monitoring systems. When applied to traffic control systems, it is possible to monitor traffic conditions in real time and improve the accuracy of time synchronization between camera sensors and lidar sensors, which can be used for accurate data collection and analysis, accident prevention, and optimization of traffic flow. Furthermore, when applied to autonomous vehicles, the recognition system of the autonomous vehicle can fuse data from camera sensors and lidar sensors to recognize the driving environment more accurately and robustly, and for this purpose, time synchronization between sensors according to the present invention can be applied to improve the safety of the recognition system. In the field of security monitoring systems, accurate recognition technology for camera sensors and lidar sensors used for monitoring in public places and security facilities is required, and for this purpose, time synchronization between sensors according to the present invention can be applied to improve the quality and reliability of data, which can be used in crime prevention and emergency accident detection systems.

[0127] On the other hand, a time synchronization system and method between a camera sensor and a lidar sensor according to one embodiment of the present invention can be implemented in the form of program instructions that can be performed by various means of electronically processing information and can be recorded on a storage medium. The storage medium may include program instructions, data files, data structures, etc., either alone or in combination.

[0128] The program instructions recorded on the storage medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art in the software field. Examples of storage mediums include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code executable by devices that process information electronically using interpreters, such as computers.

[0129] As described above, the present invention illustrates specific components and other details, as well as limited embodiments, with reference to the drawings. These are provided to facilitate a more general understanding of the present invention, and the present invention is not limited to the aforementioned embodiments. Any person with ordinary skill in the art to which the present invention belongs can make various modifications and variations from this description.

[0130] Therefore, the concept of the present invention should not be limited to the embodiments described above. Not only the claims described later, but also all variations equivalent to or comparable to these claims can be said to fall within the scope of the concept of the present invention. [Explanation of Symbols]

[0131] 100 First detection unit 200 Second detection unit 300 1st estimation part 400 Second estimation part 500 Object Judgment Department 600 Synchronization Processing Unit

Claims

1. A first detection unit analyzes input image data via a pre-stored first data processing algorithm to detect moving object data and groups the detected moving object data, A second detection unit analyzes point cloud data input via a pre-stored second data processing algorithm to detect moving object data and performs crowding of the detected moving object data. The first detection unit calculates the center point position of each grouped moving object data, and uses this to estimate motion-related information for each moving object data; The second detection unit projects the data of each moving object, which has been collected by the second detection unit, onto the image pixel coordinate system and performs a coordinate system transformation. After that, the second estimation unit calculates the center point position based on the position coordinates in the transformed image pixel coordinate system and uses this to estimate the motion-related information of each moving object data. An object determination unit analyzes the center point position of each moving object data obtained by the first estimation unit and the center point position based on the position coordinates in the image pixel coordinate system converted by the second estimation unit, using the image pixel coordinate system as a reference, and determines that the moving object data is the same. A time synchronization system between a camera sensor and a lidar sensor, which includes a synchronization processing unit that compares motion-related information of each moving object data determined to be the same moving object data, calculates a time offset, and performs time synchronization between the camera sensor and the lidar sensor using the calculated time offset.

2. The aforementioned motion-related information is, A time synchronization system between a camera sensor and a lidar sensor according to claim 1, including information on the speed and acceleration of a moving object.

3. The synchronization processing unit is, A time synchronization system for a camera sensor and a lidar sensor according to claim 2, which compares the velocity information of each moving object data determined to be the same moving object data and calculates the time offset of the camera sensor and lidar sensor corresponding to each image data and point cloud data.

4. The synchronization processing unit is, A time synchronization system for a camera sensor and a lidar sensor according to claim 3, wherein the time offset calculated is applied to the velocity information of the moving object data obtained from the point cloud data, and the parallel translation vector of each moving object data is corrected to synchronize the time between the camera sensor and the lidar sensor.

5. A time synchronization system for a camera sensor and a lidar sensor, in which each step is performed by a calculation processing means, and a method for time synchronization between a camera sensor and a lidar sensor, In the first detection unit, the first detection step (S100) involves analyzing the image data from the input camera sensor via a pre-stored first data processing algorithm to detect moving object data and then grouping the detected moving object data, In the second detection unit, a second detection step (S200) is performed in which point cloud data from the input lidar sensor is analyzed via a pre-stored second data processing algorithm to detect moving object data, and the detected moving object data is aggregated. In the first estimation unit, the center point position of each moving object data that has been clustered by the first detection step (S100) is calculated, and using this, the first estimation step (S300) estimates the motion-related information of each moving object data, In the second estimation unit, a coordinate transformation step (S400) is performed by projecting the data of each moving object that has been grouped by the second detection step (S200) onto the image pixel coordinate system and performing a coordinate system transformation, In the second estimation unit, the center point position is calculated based on the position coordinates in the image pixel coordinate system transformed by the coordinate transformation step (S400), and this is used to estimate motion-related information for each moving object data in the second estimation step (S500), In the object determination unit, the center point position for each moving object data calculated in the first detection step (S100) in the first estimation step (S300) and the center point position for each moving object data calculated in the second detection step (S200) in the second estimation step (S500) are analyzed based on the image pixel coordinate system, and the object determination step (S600) determines that the moving object data is the same. In the synchronization processing unit, an offset calculation step (S700) is performed to calculate a time offset by comparing the motion-related information of each moving object data that has been determined to be the same moving object data in the object determination step (S600), A method for synchronizing the time between a camera sensor and a lidar sensor, comprising: a synchronization processing step (S800) in the synchronization processing unit, which performs time synchronization between the camera sensor and the lidar sensor using the time offset calculated in the offset calculation step (S700).

6. The aforementioned object determination step (S600) is, A method for synchronizing a camera sensor and a lidar sensor over time, according to claim 5, wherein the center point position for each moving object data calculated in the first detection step (S100) in the first estimation step (S300) and the center point position for each moving object data calculated in the second detection step (S200) in the second estimation step (S400) are analyzed with respect to the image pixel coordinate system, and the pair of moving object data corresponding to the most adjacent center points are determined to be the same moving object data.

7. The motion-related information obtained from the first estimation step (S300) and the second estimation step (S500) is as follows: A method for time-synchronizing a camera sensor and a lidar sensor according to claim 5, including information on the speed and acceleration of a moving object.

8. The offset calculation step (S700) is, The method for synchronizing a camera sensor and a lidar sensor according to claim 7, comprising comparing the velocity information of each moving object data determined to be the same moving object data in the object determination step (S600), and calculating the time offset of the camera sensor and lidar sensor corresponding to each image data and point cloud data.

9. The synchronization processing step (S800) is, A method for synchronizing a camera sensor and a lidar sensor according to claim 8, wherein the time offset calculated in the offset calculation step (S700) is applied to the velocity information of the moving object data from the point cloud data, and the translation vector of each moving object data is corrected to synchronize the time between the camera sensor and the lidar sensor.

Images