Semantic segmentation-based external compensation device and method

Abstract

An external correction device based on semantic segmentation according to an embodiment includes a communication module; a memory for storing at least one instruction; and a processor, wherein the at least one instruction is executed by the processor, the external correction device collects image data from a camera and collects point cloud data from a lidar, performs semantic segmentation based on the collected image and point cloud data, aligns the semantic segmentation results obtained from the camera and lidar to detect the same object, extracts extrinsic parameters for determining the relative position and direction between the camera and lidar using the detection results, and transforms the data collected from the camera and lidar to have the same coordinate system based on the extrinsic parameters.

Description

Technology Field

[0001] The technical concept of the present disclosure relates to an extrinsic calibration device and method based on semantic segmentation, and more specifically, to an device and method for automating an extrinsic calibration process and a time synchronization process using a machine learning (ML) technique utilizing semantic information. Background Technology

[0002] Unless otherwise indicated in this specification, the contents described in this section are not prior art for the claims of this application, and are not to be recognized as prior art simply because they are included in this section.

[0003] Sensor fusion is a technology that combines information from different sensors to simultaneously acquire various characteristics of an object, as well as 3D positional and motion information. Cameras can acquire rich 2D color information of objects, and based on this, they can recognize the type of object and various color information. However, cameras are unsuitable for acquiring information regarding distance and object movement. On the other hand, LiDAR sensors are suitable for determining the precise 3D position and structure of a specific object, but their resolution is limited, making them unsuitable for acquiring external information such as color.

[0004] Figure 1 is a diagram showing the result of sensor information fusion in the prior art. To use multiple sensors together, it is necessary to accurately know where each sensor is spatially located and in what direction (angle) it is positioned. To determine the position and orientation of the sensors, the relative positions and orientations between each sensor must be adjusted through 3D position translation and rotation. At this time, the values ​​(parameters) required for the translation must be accurately known in order to correctly combine the sensor data.

[0005] For example, autonomous vehicles perceive their surrounding environment using various sensors such as LiDAR, cameras, and IMUs (Inertial Measurement Units). LiDAR provides distance data, cameras provide color and shape information, and IMUs measure the vehicle's tilt or rotation information. However, since each sensor has a different position and orientation, data fusion is difficult because the data does not align unless the precise position and rotation relationships are corrected (Extrinsic Calibration).

[0006] Furthermore, since each sensor collects data in a physically different way, slight time differences may occur even when recording the same event. For example, assuming a camera captures at 30 frames per second (30Hz) and a LiDAR collects data 10 times per second (10Hz), the two sensors do not collect data at the same time but rather experience slight time differences. Additionally, synchronization is not perfect due to communication speeds between sensors or internal system processing delays. Consequently, to perform sensor fusion, the position and orientation of each sensor must be accurately calibrated (Extrinsic Calibration). However, since there may be time differences (timing errors) in the collected data from the sensors, both issues regarding position calibration (spatial alignment) and time synchronization (time alignment) must be resolved.

[0007] Meanwhile, sensors have traditionally been calibrated manually using artificial markers such as checkerboards. However, checkerboard-based calibration methods require direct installation, and the manual nature of the work increases time and costs. Furthermore, the checkerboard method is difficult to use in outdoor and complex environments, and recalibration is required if the sensor position changes. Additionally, it is difficult to integrate with automatic calibration systems. Prior art literature

[0008] 1. Korean Patent Publication No. 10-2025-0012894 (January 31, 2025) 2. Korean Patent Publication No. 10-2024-0037122 (March 21, 2024) The problem to be solved

[0009] The semantic segmentation-based extrinsic calibration device and method according to the embodiment perform extrinsic calibration between a camera and a lidar using semantic segmentation, thereby enabling the two sensors to recognize and match the same object and correct its relative position and orientation.

[0010] In addition, the semantic segmentation-based external correction device and method according to the embodiment perform correction by utilizing collected data including images and point clouds without using artificial markers.

[0011] In addition, in the embodiment, data obtained from different sensors (e.g., camera, LiDAR) is analyzed using semantic segmentation to identify identical objects and then align them.

[0012] In addition, in the embodiment, objects such as buildings, vehicles, and people are identified in each data through semantic segmentation, and different regions are distinguished by color to enable the identification of the same object in camera data and LiDAR data.

[0013] In addition, in the embodiment, during the matching process, the semantic segmentation results obtained from the camera and the lidar are aligned to find the same object, thereby extracting extrinsic parameters, which are information for determining the relative position and orientation between the sensors.

[0014] However, the problem to be solved according to one embodiment is not limited only to that mentioned above. means of solving the problem

[0015] An external correction device based on semantic segmentation according to an embodiment includes a communication module; a memory for storing at least one instruction; and a processor, wherein the at least one instruction is executed by the processor, the external correction device collects image data from a camera and collects point cloud data from a lidar, performs semantic segmentation based on the collected image and point cloud data, aligns the semantic segmentation results obtained from the camera and lidar to detect the same object, extracts extrinsic parameters for determining the relative position and direction between the camera and lidar using the detection results, transforms the data collected from the camera and lidar to have the same coordinate system based on the extrinsic parameters, and the extrinsic parameters may include a rotation matrix (R) and a translation vector (t) of the camera and lidar.

[0016] In addition, the external calibration device evaluates the fitness during the external calibration process between the camera and the LiDAR, and the fitness may be the accuracy of the estimated extrinsic parameter.

[0017] In addition, the external correction device measures the alignment accuracy of the LiDAR point cloud over the camera image to calculate the suitability, and the suitability

[0018] Mathematical formula 1

[0019]

[0020] It is calculated through the above mathematical formula 1, where f1(x) is the number of LiDAR points existing in the camera image coordinate system, f2(x) is the number of semantic information matches between the LiDAR and the camera, and α and β in the above mathematical formula 1 may be weights.

[0021] In addition, the external correction device uses f1(x) of Equation 1

[0022] Mathematical formula 2

[0023]

[0024] Calculated through,

[0025] The above f2(x) is mathematical formula 3

[0026]

[0027] It can be calculated through.

[0028] In addition, the external correction parameter includes a rotation parameter, the rotation parameter includes roll rotation (rot_x) and yaw rotation (rot_z) values, and the external correction device can detect an optimal peak at a specific rotation value.

[0029] In addition, the external correction device can calculate the optimal solution by using a nonlinear search function including a Genetic Algorithm (GA) and an article Swarm Optimization (PSO).

[0030] In addition, the external correction device can perform time synchronization by comparing the time stamps of the camera and the lidar to match the closest data.

[0031] In addition, the external correction device combines semantic segmentation and object tracking algorithms to perform correction even in moving environments.

[0032] In addition, the external correction device can perform corrections that take into account object size (vehicle, pedestrian) or speed changes (stop, acceleration, rotation) through adaptive object-based correction.

[0033] In addition, the external correction device can learn the features of the camera and lidar to automatically search for the optimal correction point.

[0034] In addition, the external correction device performs reinforcement learning-based correction optimization and can automate reinforcement learning-based correction optimization. Effects of the invention

[0035] The semantic segmentation-based external correction device and method according to the embodiment can save costs and time by automating the correction process through the use of semantic segmentation to automatically perform correction between the camera and the lidar.

[0036] In addition, through the embodiment, the same object (building, vehicle, person, etc.) is recognized and aligned in the camera and lidar, so that the data of the two sensors is precisely matched, thereby increasing the accuracy of data alignment between sensors.

[0037] In addition, through the embodiments, sensor fusion performance can be improved in autonomous vehicles, robot vision, and AR / VR systems.

[0038] In addition, while conventional methods required placing artificial markers such as checkerboards or specific patterns in the environment, the embodiment provides a more practical sensor calibration method by using semantic segmentation in the actual environment (natural environment) for calibration.

[0039] In addition, the calibration method provided in the embodiment can be utilized in various environments requiring sensor fusion, such as autonomous driving, drones, and robots, and enables calibration in various environments, including roads, indoor environments, and factory automation.

[0040] In addition, the embodiment improves the object recognition accuracy of autonomous vehicles and the path planning reliability of robots by using semantic segmentation through machine learning to align sensor data more precisely.

[0041] In addition, the embodiment enables the improvement of the performance of the artificial intelligence model by providing accurate sensor data to enhance the reliability of the artificial intelligence training data.

[0042] The effects obtainable from the exemplary embodiments of the present disclosure are not limited to those mentioned above, and other unmentioned effects can be clearly derived and understood by those skilled in the art to which the exemplary embodiments of the present disclosure belong from the description below. That is, unintended effects resulting from the implementation of the exemplary embodiments of the present disclosure can also be derived by those skilled in the art from the exemplary embodiments of the present disclosure. Brief explanation of the drawing

[0043] Figure 1 is a diagram showing the result of conventional sensor information fusion. FIG. 2 is a drawing for explaining the concept of extrinsic calibration performed in an embodiment and a conventional checkerboard-based calibration method. FIG. 3 is a drawing for explaining the correction process of a semantic division-based external correction device according to an embodiment. FIG. 4 is a diagram comparing the accuracy and error of extrinsic calibration using semantic partitioning according to an embodiment. The graph in Fig. 5 is a goodness-of-fit curve, where the X-axis represents the change in R and T values ​​(degree of correction) and the Y-axis represents the fitness value. Figures 6a, 6b, and 6c represent three semantic segmentation images. Figure 7 is a diagram illustrating the time synchronization problem that occurs when fusing camera and LiDAR data. FIG. 8 is a diagram illustrating a method to solve the time synchronization problem that occurs when fusing camera and LiDAR data. FIG. 9 is a block diagram of an external correction device according to an embodiment. FIG. 10 is a drawing showing an external correction process according to an embodiment. Specific details for implementing the invention

[0044] Hereinafter, various embodiments of the present disclosure are described in conjunction with the accompanying drawings. As various embodiments of the present disclosure may be subject to various modifications and may have various embodiments, specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the various embodiments of the present disclosure to specific forms, and it should be understood that they include all modifications and / or equivalents and substitutions that fall within the spirit and scope of the various embodiments of the present disclosure. In relation to the description of the drawings, similar reference numerals have been used for similar components.

[0045] In various embodiments of the present disclosure, terms such as “comprising” or “having” are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0046] In various embodiments of the present disclosure, expressions such as “or” include any and all combinations of the words listed together. For example, “A or B” may include A, may include B, or may include both A and B.

[0047] Expressions such as "first," "second," "first," or "second" used in various embodiments of the present disclosure may modify various components of the various embodiments, but do not limit such components. For example, such expressions do not limit the order and / or importance of such components and may be used to distinguish one component from another.

[0048] When it is mentioned that a component is "connected" or "joined" to another component, it should be understood that the component may be directly connected or joined to the other component, but that a new component may also exist between the component and the other component.

[0049] In the embodiments of the present disclosure, terms such as "module," "unit," "part," etc. are used to refer to a component that performs at least one function or operation, and such component may be implemented in hardware or software, or in a combination of hardware and software. Additionally, a plurality of "modules," "units," "parts," etc. may be integrated into at least one module or chip and implemented as at least one processor, except where each needs to be implemented in specific individual hardware.

[0050] Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in the various embodiments of the present disclosure.

[0051] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0052] FIG. 2 is a diagram illustrating the concept of extrinsic calibration performed in the embodiment and a conventional checkerboard-based calibration method. Referring to FIG. 2, the embodiment performs an extrinsic calibration process between a camera and a LiDAR. This corrects the position and orientation differences between different sensors. Extrinsic calibration is an algorithm that estimates and corrects the three-dimensional position and angle differences (R, T) between different sensors (camera, LiDAR, etc.). In the embodiment, extrinsic calibration enables the data obtained from each sensor to be converted into the same coordinate system and fused. The rotation matrix (R, Rotation) shown in FIG. 2 corrects the angle difference where the sensors face different directions, and the translation matrix (T, Translation) corrects the distance difference where the sensors are mounted at different locations. Meanwhile, the checkerboard pattern in FIG. 2 serves as a calibration target; conventionally, a checkerboard pattern is utilized to enable the camera and LiDAR to recognize the same reference point.

[0053] In the embodiment, the LiDAR collects 3D point cloud data, and the camera collects 2D image data. Subsequently, the extrinsic parameters R and T are adjusted by comparing whether the two sensors are looking at the same position relative to the checkerboard. Through this, a correction is performed to align the camera and LiDAR data into the same coordinate system. However, since the artificial marker (checkerboard) requires additional installation and is difficult to automate, the embodiment proposes an automatic correction method based on semantic segmentation.

[0054] FIG. 3 is a diagram illustrating the calibration process of a semantic segmentation-based external calibration device according to an embodiment. Referring to FIG. 3, the semantic segmentation-based external calibration device according to the embodiment performs an external calibration process by utilizing semantic segmentation. In the embodiment, the external calibration device performs calibration by utilizing data including images and point clouds without using artificial markers. In the embodiment, identical objects are identified and aligned from data obtained from different sensors (camera, LiDAR) using semantic segmentation. To this end, an extrinsic calibration process is performed between the image, which is camera data, and the point cloud, which is LiDAR data.

[0055] In the embodiment, original data is acquired for external correction. The original data may include image data acquired from a general camera and point clouds, which are 3D point cloud data collected from a LiDAR. In the embodiment, the LiDAR collects 3D point cloud data, and the camera collects 2D image data. Subsequently, the external correction device performs semantic segmentation. In the embodiment, semantic segmentation includes a process of identifying objects such as buildings, vehicles, and people in each data set and distinguishing different regions by color. Subsequently, the same object can be found in the camera data and the LiDAR data. Additionally, the external correction device detects the same object by aligning the semantic segmentation results obtained from the camera and LiDAR through a matching process. In the embodiment, extrinsic parameters, which are information necessary to determine the relative position and orientation between sensors including the camera and LiDAR, are extracted during the object detection process.

[0056] In the embodiment, the external correction device calculates the position and orientation (R, t) of the camera and lidar as shown in FIG. 3 to determine the extrinsic correction parameter. In the embodiment, the rotation matrix (R) and the translation vector (t) are detected and transformed so that the data of the two sensors have the same coordinate system.

[0057] In the embodiment, the rotation matrix R (Rotation Matrix) may include a 3x3 matrix that rotates one coordinate system to another coordinate system, and the translation vector t (Translation Vector) may include a 3x1 vector that translates one coordinate system to another coordinate system. The external correction device according to the embodiment acquires the two parameters mentioned above to enable transformation of the coordinate system of a specific sensor to the coordinate system of another sensor. For example, when the same object (building, road, vehicle, etc.) is detected by the camera and the lidar, the same object is detected by each sensor using semantic segmentation. Subsequently, R and t are calculated by matching the identical object information of the two sensors. At this time, the external correction device can detect the same object recognized by the camera and the lidar using a deep learning-based object matching algorithm (e.g., YOLO, Mask R-CNN) and a feature matching algorithm (e.g., SIFT, ORB, SURF).

[0058] In the embodiment, calibration is enabled without conventional artificial markers such as a checkerboard, and sensor calibration can be performed in a real-world environment. Additionally, camera and LiDAR data can be effectively combined and utilized for applications such as autonomous driving and robot vision.

[0059] FIG. 4 is a diagram comparing the accuracy and error of extrinsic calibration using semantic segmentation according to an embodiment. With reference to FIG. 4, the process of estimating extrinsic calibration parameters (R, T) in extrinsic calibration will be explained below. FIG. 4 compares a successful case and a failed case in the process of correctly aligning camera and lidar data.

[0060] In the embodiment, since the camera and LiDAR collect data at different positions and angles, the two sets of data must be accurately aligned (corrected). At this time, the external calibration device according to the embodiment estimates the position (Translation, T) and direction (Rotation, R) values ​​and performs an extrinsic calibration process to correctly project the LiDAR point cloud onto the camera image.

[0061] The successful calibration (result of successful parameter estimation) illustrated in Fig. 4a shows that when accurate R and T values ​​are estimated, the 3D points obtained from the LiDAR are correctly aligned on the camera image. As shown in Fig. 4a, when accurate parameter values ​​are estimated, the colors of each object (road, tree, building, etc.) match between the image and the semantic segmentation result, and the LiDAR points are also correctly positioned. On the other hand, Fig. 4b illustrates a calibration failure (result of failed parameter estimation). As shown in Fig. 4b, if inaccurate R and T values ​​are used due to a failed parameter estimation, the LiDAR points are matched to incorrect positions on the image. Consequently, the boundaries of each object are misaligned, and the colors between the semantically segmented classes do not match. In other words, the actual location and the LiDAR points are misaligned, causing the fusion of the camera and LiDAR data to fail.

[0062] Accordingly, the external correction device according to the embodiment ensures that the LiDAR point is accurately projected onto the image through accurate extrinsic parameter estimation. In the embodiment, the extrinsic parameters include position (Translation, T) and direction (Rotation, R) values. In the embodiment, accurate extrinsic parameters are calculated to correctly match the data of the camera and LiDAR, and the semantic segmentation information of each object, including buildings, roads, trees, etc., is also matched. Through this, the camera-LiDAR sensor fusion accuracy is improved, and correction can be performed without an artificial marker (Checkerboard) through automatic correction using semantic segmentation.

[0063] FIG. 5 is a diagram illustrating the fitness evaluation process of external correction using semantic segmentation according to an embodiment. Referring to FIG. 5, in the embodiment, the external correction device evaluates fitness during the external correction process between the camera and the LiDAR. In the embodiment, the accuracy of the estimated extrinsic parameters R and T can be mathematically evaluated through fitness. In the embodiment, fitness is a measure that evaluates whether the extrinsic parameters, including the R (rotation) and T (translation) values ​​in the external correction, are accurate, and is a value that measures how accurately the LiDAR point cloud is aligned on the camera image.

[0064] In the embodiment, the goodness-of-fit function f(x) is calculated to detect cases where the R and T values ​​are optimally corrected. In the embodiment, the goodness-of-fit is the weighted sum of the number of point clouds projected onto the image and the class matching probabilities of the projected point clouds. In the embodiment, the goodness-of-fit function can be calculated through Equation 1.

[0065] Mathematical formula 1

[0066]

[0067] In mathematical formula 1, f1(x) is the number of LiDAR points existing in the camera image coordinate system, and f2(x) is the number of times the semantic information of the LiDAR and the camera matches. That is, in the embodiment, the suitability is evaluated by summing the number of LiDAR points projected onto the camera image and the degree to which each point matches semantically.

[0068] f1(x) of Equation 1 can be calculated through Equation 2, and f2(x) of Equation 1 can be calculated through Equation 3.

[0069] Mathematical formula 2

[0070]

[0071] Mathematical formula 3

[0072]

[0073] Referring to Figures 5a, 5b, and 5c, Figure 5a shows a diagram where the LiDAR points are correctly aligned, and it can be confirmed that the color areas match. Figure 5b shows an optimal state, where the LiDAR points are aligned most accurately. Figure 5c shows an incorrect correction, where it can be confirmed that the LiDAR points are misaligned and the color areas are misaligned.

[0074] The graph in Fig. 5 is a fitness curve, where the X-axis represents the change in R and T values ​​(degree of correction) and the Y-axis represents the fitness value. The peak of the graph represents the optimal correction value. Referring to Fig. 5, it can be seen that the fitness score decreases as the value deviates from the peak, that is, as the LiDAR point becomes misaligned.

[0075] In the embodiment, the external calibration device enables automatic calibration based on mathematical optimization through the degree of fit, thereby improving calibration accuracy. Additionally, by utilizing semantic information, it enhances the reliability of sensor fusion and allows for the quantitative evaluation of calibration errors, enabling highly reliable sensor alignment.

[0076] FIG. 6 is a diagram showing the results of the analysis of the influence of rotation parameters rot_x and rot_z in external correction according to an embodiment. FIG. 6 illustrates the process of analyzing the influence of rotation parameters (rot_x, rot_z) on the correction result during the external correction process between a camera and a lidar.

[0077] Rotation correction is a process of correcting the relative position (T) and rotation (R) of the camera and LiDAR. In particular, if the rot_x (roll rotation) and rot_z (yaw rotation) values ​​are incorrectly set, the camera and LiDAR data become misaligned, leading to inaccurate object matching. rot_x, representing X-axis rotation (Roll), indicates the degree to which the camera or LiDAR is tilted left or right. rot_z, representing Z-axis rotation (Yaw), indicates the degree to which the camera or LiDAR is rotated horizontally. If the two rotation values ​​mentioned above—X-axis rotation (Roll) and Z-axis rotation (Yaw)—are incorrect, the LiDAR point cloud is aligned to an incorrect position in the image.

[0078] Figures 6a, 6b, and 6c represent three semantic segmentation images. Figure 6a shows a color segmentation area misaligned with incorrect rot_x and rot_z values, while Figure 6b shows a color segmentation area correctly aligned with optimal rot_x and rot_z values. Figure 6c shows a color segmentation area misaligned with yet another incorrect rot_x and rot_z value. In other words, the degree of sensor data alignment varies significantly depending on the value of the rotation parameter (R).

[0079] The graph in Fig. 6 represents the change in fitness, where the X-axis represents the change in rot_x and rot_z values, and the Y-axis represents the fitness value. The peak of the graph shown in Fig. 6 represents the optimal correction value, at which point the camera and LiDAR data are best aligned. As the curve drops (moves further to the left or right), it indicates that the correction is incorrect. In other words, the fitness reaches an optimal peak at a specific rotation value, and accurate correction is possible only by finding this value. In the embodiment, external correction is automatically optimized to reduce the need for manual human intervention. Furthermore, the influence of rotation values ​​(rot_x, rot_z) can be quantitatively evaluated, and the precision of LiDAR and camera sensor fusion can be improved, thereby increasing object detection accuracy. In the embodiment, appropriate rot_x and rot_z values ​​are detected based on the change in rot_x and rot_z to ensure that the camera and LiDAR data are accurately aligned.

[0080] Since the fitness calculated during the parameter tracking process is a nonlinear function, a nonlinear search function is used in the embodiments. Additionally, the Genetic Algorithm (GA) and Particle Swarm Optimization (PSO) may be applied in the embodiments. The parameter tracking performed in the embodiments is a process of optimally detecting the relative position (Translation, T) and rotation (Rotation, R) between sensors during external correction. To this end, the embodiments calculate a fitness function to detect the R and T values ​​that yield the highest fitness. However, since this fitness function is nonlinear rather than linear, the embodiments utilize nonlinear optimization techniques instead of general linear search methods. In the embodiments, the fitness function measures how accurately the data from the camera and LiDAR are aligned. However, when the relative positions (R, T) of the camera and LiDAR change, the fitness value does not change linearly but exhibits a complex pattern. In other words, since the fitness function is not a simple linear function but a nonlinear function, the optimal point cannot be easily found using simple methods (e.g., Gradient Descent). To solve this, nonlinear optimization techniques are required, and in the examples, Genetic Algorithm (GA) and Particle Swarm Optimization (PSO) techniques can be used.

[0081] The Genetic Algorithm (GA) is an optimization technique that mimics the process of natural evolution. It explores multiple solutions simultaneously and finds progressively better solutions through genetic operations (crossing, mutation, and selection). Genetic algorithms have a high probability of finding optimal values ​​even in complex non-linear functions.

[0082] In the example, multiple initial R and T values ​​are randomly generated for the application of GA. This is the process of generating an initial population. Subsequently, a fitness function is calculated for each R and T value. In the example, the most suitable R and T values ​​are selected to generate a new generation (individuals). For example, the processes of selection, crossover, and mutation can be repeated. In the example, the optimal R and T values ​​are derived after the repetition. In the example, various solutions are explored simultaneously through the genetic algorithm to prevent falling into a local optimum.

[0083] In addition, the embodiment can calculate the optimal solution through the article swarm optimization (PSO) process. Article swarm optimization is a method in which multiple solutions (particles) cooperate to find the optimal value; each particle searches for the optimal solution by moving slightly while referring to its current position and surrounding optimal values. Article swarm optimization is mathematically simple yet enables powerful performance in non-linear functions.

[0084] To this end, particle swarm optimization performs a particle initialization process that randomly generates multiple initial R and T values. Subsequently, each particle calculates a fitness value and moves to a new position by comparing its own optimal value with that of its neighbors. In the example, the process converges to increasingly accurate R and T values ​​through iteration. Particle swarm optimization enables the rapid finding of optimal solutions even for complex functions with minimal computational effort.

[0085] Figure 7 is a diagram illustrating the time synchronization problem that occurs when fusing camera and lidar data. Since different sensors may collect data at different times, a process to synchronize them is required.

[0086] Different sensors, such as cameras and LiDAR, have varying frame rates (FPS), and data is often recorded at different timestamps due to internal and external system differences. If these differences are not compensated for, errors caused by time discrepancies may occur when fusing sensor data. Therefore, to precisely combine camera and LiDAR data, sensor synchronization (Time Synchronization), which synchronizes data based on the same time standard, is required.

[0087] Sensors (cameras, LiDAR) observe the same object (person), but data may be collected at different times. As illustrated in Fig. 7, since cameras capture frames at a specific moment and LiDAR collects data at specific intervals, it is important to locate data from the same point in time. Specifically, because cameras capture images (frames) at regular intervals (e.g., 30 FPS), data is collected at equal intervals. On the other hand, LiDAR generally collects data at a lower rate than cameras (e.g., 10 FPS). Therefore, data is recorded at different intervals than that of cameras. Since camera and LiDAR data are not recorded at exactly the same point in time, a calibration process is required to match the data.

[0088] In the embodiment, the aforementioned problem can be solved through a time synchronization technique. In the embodiment, data is matched through timestamp correction. The external correction device according to the embodiment compares the timestamps (time information) of the camera and the lidar to match the data at the nearest point in time.

[0089] In addition, interpolation is used in the embodiments. In the embodiments, Linear Interpolation or Spline Interpolation techniques may be used to predict data at a specific point in time. Furthermore, in the embodiments, GPS or PTP (Precision Time Protocol) is used to hardware-configure all sensors to operate at the same time standard.

[0090] In the embodiment, the image solves the time synchronization problem between the camera and the lidar, thereby synchronizing sensors that collect data at different intervals to improve object recognition accuracy and enable precise sensor fusion.

[0091] FIG. 8 is a diagram illustrating a method for solving the time synchronization problem that occurs when fusing camera and LiDAR data. Referring to FIG. 8, since the data collection times differ for each sensor, the process of synchronizing them to match data at the same point in time is visually illustrated. As shown in FIG. 8, the camera captures frames (Tc) at regular intervals (e.g., 30 FPS). That is, it collects data at equal intervals and stores video frames. In the embodiment, the LiDAR collects data at a lower frequency than the camera (e.g., 10 FPS). Therefore, data is recorded less frequently by the LiDAR than by the camera. Even if the camera and LiDAR detect the same object, there exists a time difference (Tc - TL) of the collected data, which is Dt. In the embodiment, the time difference (Dt) must be calculated and corrected so that the same object can be accurately aligned at the same point in time. In the embodiment, an external correction device calculates Dt (time difference) and performs the process of synchronizing LiDAR data with camera frames.

[0092] In the embodiment, LiDAR data is projected onto a camera image (Projecting LiDAR Point Cloud onto Camera), and the LiDAR point clouds are converted into the camera coordinate system and projected onto the image. However, if the time of the projected data differs, correction is required.

[0093] In the embodiment, the external correction device performs motion-based correction using object movement information and calculates and corrects the actual distance (Dt) traveled by moving objects (e.g., cars, pedestrians, etc.). This enables matching objects located at the same position at the same point in time. Additionally, in the embodiment, the closest data is matched by comparing the timestamps of the camera and LiDAR through timestamp synchronization and interpolation. In the embodiment, linear interpolation or spline interpolation techniques may be used to predict data at a specific point in time.

[0094] In addition, in the embodiment, when the external correction device estimates the camera's extrinsic parameters, more accurate results can be obtained if it utilizes a fixed object (e.g., a wall, floor, building, etc.) obtained through semantic analysis. In the embodiment, by utilizing information on the fixed object obtained from the semantic result, extrinsic parameters including the camera's position and orientation can be adjusted more precisely.

[0095] In addition, in the embodiment, the external calibration device combines semantic segmentation and object tracking algorithms to enable calibration even in moving environments. For example, the external calibration device can track the same vehicle or pedestrian using both a camera and a lidar, and perform calibration based on the movement of the object. To this end, in the embodiment, the external calibration device performs sensor synchronization and calibration by combining spatio-temporal calibration. For example, the external calibration device analyzes the continuous movement of a moving object to simultaneously perform timestamp offset and extrinsic calibration between sensors. In the embodiment, optimal time synchronization is performed by comparing and analyzing data sampling cycles to correct the timestamp offset between sensors. For example, the external calibration device compares timestamps between sensors based on the same event and applies a calibration model to correct the time error. In the embodiment, the same event recognized by the camera and lidar includes, but is not limited to, feature points and object detection. Furthermore, the calibration model includes, but is not limited to, linear interpolation, NCC, and Kalman filters.

[0096] In the embodiment, the external correction device first aligns the data of the two sensors based on a specific event (object detection, motion change, etc.) by correcting time errors through a correction model. Subsequently, it detects common feature points (e.g., pedestrian, vehicle, sign) to calculate the time difference (Timestamp Offset) between the camera and LiDAR data, and can synchronize by detecting the nearest timestamp that matches the feature points.

[0097] In the embodiment, the external correction device may utilize Normalized Cross-Correlation (NCC), identical event feature matching, or signal similarity optimization algorithms to detect timestamps with maximum correlation by comparing signal patterns of the camera and LiDAR. The aforementioned methods include a correction procedure to find the optimal timestamp that minimizes the temporal difference between the two sensors. In the embodiment, the external correction device calculates the optimal time synchronization offset (Δt) based on visual / reflective changes occurring in the same scene using a Normalized Cross-Correlation (NCC)-based timestamp detection method. To this end, signal patterns of the camera and LiDAR are collected, and specific events (changes in pixel intensity, edge changes, etc.) are extracted from the camera:frame sequence. Additionally, changes in intensity of the LiDAR scan are extracted. In the embodiment, specifically, changes in the position of objects with reflectivity above a certain level can be extracted. Subsequently, the dimensions of the camera and LiDAR data are normalized and transformed to enable comparison between patterns. For example, pixel intensity is normalized to [0, 1] and lidar intensity is normalized to [0, 1]. Then, an external correction device performs cross-correlation between the camera signal pattern and the lidar signal pattern with different time delays (Δt). After that, a timestamp that maximizes the normalized cross-correlation value is detected, and the lidar and camera are synchronized with the detected timestamp.

[0098] In addition, the external correction device matches the same object (e.g., vehicle, sign, pedestrian, etc.) in the camera and the lidar, and calculates the time difference (Δt) of the feature points by comparing the timestamps of each feature point. Subsequently, the external correction device can correct the time difference between the camera and the lidar by the time difference (Δt) of the feature points.

[0099] In addition, the external correction device corrects the relative positions (R, t) of the camera and lidar based on the path of the moving object. Subsequently, it automatically calculates the coordinate system difference between the sensors by utilizing information on the continuous position change of the object. In the embodiment, R (rotation matrix) and t (movement vector) are optimized by utilizing SVD (Singular Value Decomposition) or ICP (Iterative Closest Point).

[0100] ICP is an iterative optimization algorithm that directly aligns two 3D point clouds. The external correction device uses ICP to optimize R (rotation matrix) and t (translation vector) when the camera and LiDAR point clouds are not initially aligned. In the embodiment, if there are no initially estimated R and t, approximate R and t are set arbitrarily. Additionally, the external correction device can apply ICP for fine correction after performing initial alignment using the SVD method. In the embodiment, the external correction device detects the point closest to the LiDAR point cloud Q for the point cloud P of the image acquired from the camera and matches the two detected points. Subsequently, the rotation and translation matrices are updated by applying the SVD method using the matched points. In the embodiment, the updated rotation matrix (R) and t (translation vector) are applied to all points to perform the transformation. In the embodiment, this process is repeated until the amount of change falls below a certain threshold. Through this, the external correction device can automatically align the coordinate systems of the camera and LiDAR with respect to the moving object, and can perform correction even in a dynamic environment without a conventional static reference point (Checkerboard).

[0101] In addition, the embodiment can precisely correct object movement by utilizing a deep learning-based Temporal Alignment network. For example, the external correction device uses an LSTM or Transformer-based network to analyze continuous object movement and dynamically adjusts the timestamp offset between sensors by utilizing information from previous and future frames. In the embodiment, the Transformer is a model that learns long-term dependencies by utilizing an attention mechanism. A model based on an attention mechanism corrects time errors by considering relationships between frames that are far apart, unlike a simple LSTM. In the embodiment, when the object trajectory (Sequence of Features) of the camera and LiDAR and the time difference patterns of past and future frames are input, an optimal time difference correction value between sensors is output. Subsequently, the model analyzes the continuous data flow of the camera and LiDAR and learns the relationship between the current frame and future frames to predict the time error. Additionally, time embeddings are added so that the model can recognize the relative time difference between frames. Afterward, the camera and LiDAR data are aligned using the predicted time difference.

[0102] In the embodiment, the external correction device converts data from past and future frames into Transformer inputs and learns patterns between sensor data to predict the optimal time difference correction value. Subsequently, it utilizes the Attention Score to select the most reliable time correction value.

[0103] In addition, the external correction device learns the temporal movement of objects to perform optimal temporal alignment between sensors. In the embodiment, correction is performed by considering the coordinate system of the sensors so that the same object can be recognized at the same coordinates. This resolves the limitation that existing correction methods are performed in static environments and make it difficult to correct moving objects (e.g., pedestrians, vehicles).

[0104] In addition, the external calibration device performs adaptive object-based calibration. In the embodiment, calibration is performed considering object size (vehicle, pedestrian) or speed changes (stationary, accelerated, rotated). In the embodiment, the external calibration device optimizes sensor calibration parameters (R, t) according to the speed and size of a specific object (pedestrian, vehicle). For example, the external calibration device can dynamically optimize calibration parameters by utilizing information on the object's size (Height, Width, Depth), speed (Vector), and acceleration (Acceleration), and can perform AI-based adaptive calibration by learning the object's motion characteristics. To this end, the external calibration device performs AI-based adaptive calibration by learning the object's motion characteristics. For example, a machine learning model analyzes the object's movement patterns to correct time differences between sensors and spatial alignment errors in real time, and automatically adjusts the optimal calibration matrix and motion vector (R, t) according to environmental changes. This enables adaptation to the dynamic changes of various moving objects and maximizes the precision of sensor data fusion.

[0105] Furthermore, the external correction device learns error models for both the camera and LiDAR sensors to automatically correct errors at specific speeds and distances. To achieve this, the external correction device collects real-time data streams from the camera and LiDAR and analyzes the object's size (Height, Width, Depth), velocity (Vector), and acceleration. Subsequently, it calculates the expected error at the current speed and distance based on the learned error models. It then detects discrepancies between the camera and LiDAR data and calculates a correction matrix. Finally, the external correction device dynamically adjusts the corrected rotation matrix and motion vector.

[0106] In addition, the external calibration device learns the features of the camera and LiDAR to automatically search for the optimal calibration point. To this end, the external calibration device performs feature alignment between sensors by combining the results of Semantic Feature Matching and Deep Feature Matching. Subsequently, it extracts the optimal calibration value by simultaneously considering the object's 3D bounding box and 2D bounding box. In the embodiment, to extract the optimal calibration value by simultaneously considering the object's 3D bounding box and 2D bounding box, the external calibration device detects the same object in the camera and LiDAR, and then automatically performs extrinsic calibration using the object's position and size information.

[0107] To this end, the external correction device receives RGB images from the camera and generates 2D bounding boxes using 2D object detection algorithms (YOLO, Faster R-CNN, etc.), and receives 3D point clouds from the LiDAR and generates 3D bounding boxes using 3D object detection algorithms (PointNet, CenterPoint, etc.). Subsequently, it matches identical objects between the 2D and 3D bounding boxes detected by the camera and LiDAR. It also performs optimal matching by comparing the object's center coordinates (Center), size (Width, Height, Depth), and attributes (Class, Confidence Score). Additionally, the external correction device generates estimated coordinates in 3D space by performing a back projection of the 2D bounding boxes from the camera coordinate system to 3D space. Then, it extracts correction values ​​(R, t) by comparing the 3D bounding boxes detected by the LiDAR with the 3D estimated coordinates transformed by the camera. Subsequently, to calculate and apply optimal correction values ​​(Extrinsic Parameters), the optimal transformation matrix (R) and translation vector (t) between the camera and the lidar are calculated based on the position, size, and rotation information of the 3D bounding box. In addition, correction values ​​that minimize the error are automatically extracted using SVD (Singular Value Decomposition) or a deep learning-based optimization network (Neural Network-based Optimization). Afterward, the external correction device applies the extracted optimal R and t values ​​to transform the data from the camera and the lidar into the same coordinate system.

[0108] In addition, the external calibration device can perform calibration optimization based on reinforcement learning. In the embodiment, calibration optimization is automated based on reinforcement learning. In the embodiment, a reinforcement learning agent learns the optimal R and t through iterative learning via simulation to increase the calibration execution speed and enable adaptation to environmental changes. The external calibration device automates calibration between sensors by utilizing a reinforcement learning-based optimization algorithm, and the reinforcement learning agent learns the optimal rotation matrix R and translation vector t by performing iterative learning in a simulation environment. The agent optimizes the reward function in a direction that minimizes calibration errors and improves the speed of sensor data alignment by applying the learned policy. Furthermore, it includes an online learning and environmental change detection mechanism to adapt to real-time environmental changes (changes in sensor position, external interference, etc.), thereby enabling faster and more precise dynamic calibration than existing calibration methods.

[0109] In the embodiment, a reinforcement learning (RL)-based agent performs iterative learning in a simulation environment and learns the optimal rotation matrix (R) and translation vector (t) between the camera and the lidar to improve the speed of correction execution and enable adaptation to various environmental changes. To this end, an external correction device randomly sets initial correction values ​​between the camera and the lidar and constructs a virtual simulation environment to generate a training dataset. Within the environment, the agent observes the coordinate transformation state of the camera and lidar data and recognizes the state requiring correction. To achieve this, a correction goal is defined and a reward function is designed so that the same object is accurately aligned in the camera and lidar after correction is performed.

[0110] In the embodiment, the reward function grants a reward when object alignment accuracy increases and enhances the reward when semantic segmentation results match. Additionally, it enables the reward to increase when alignment errors (RMSE, Chamfer Distance, etc.) are minimized.

[0111] Subsequently, agent training and the search for optimal correction values ​​are performed. In the embodiment, reinforcement learning algorithms (DQN, PPO, A3C, etc.) are utilized so that the agent learns by continuously adjusting correction values ​​(R, t) in various environments. In the embodiment, the agent is encouraged to search for optimal correction values ​​by designing the system so that it receives a higher reward as the correction becomes more accurate.

[0112] Furthermore, to apply actual data and enhance environmental adaptability, the trained agent is continuously adjusted and updated to operate in real-world environments. Additionally, in the embodiment, reinforcement learning is applied to enable automatic adaptation to changes in sensor location, vehicle movement, and indoor / outdoor environments even in new environments, and optimal correction values ​​are applied to perform real-time sensor fusion.

[0113] FIG. 9 is a block diagram of an external correction device according to an embodiment. The configuration of the external correction device (100) shown in FIG. 9 is merely a simplified example. The communication module (110) can be configured regardless of the communication mode, such as wired or wireless, and can be configured with various communication networks, such as a Personal Area Network (PAN) or a Wide Area Network (WAN). In addition, the communication module (110) can operate based on the known World Wide Web (WWW) and may utilize wireless transmission technologies used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth. For example, the communication module (110) may be responsible for transmitting and receiving data necessary to perform a technique according to an embodiment of the present disclosure.

[0114] Memory (120) may refer to any type of storage medium. For example, memory (120) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk. Such memory (120) may also constitute the database shown in FIG. 1.

[0115] The memory (120) can store at least one instruction that can be executed by the processor (130). Additionally, the memory (120) can store any form of information generated or determined by the processor (130) and any form of information received by the server (200). Additionally, the memory (120) stores various types of modules, instruction sets, or models.

[0116] The processor (130) can perform technical features according to embodiments of the present disclosure to be described below by executing at least one instruction stored in memory (120). In one embodiment, the processor (130) may be composed of at least one core and may include a processor for data analysis and / or processing, such as a central processing unit (CPU) of a computer device, a general purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU).

[0117] This processor (130) can train a neural network or model designed in a machine learning or deep learning manner. To this end, the processor (130) can perform calculations for training the neural network, such as processing input data for training, extracting features from input data, calculating errors, and updating the weights of the neural network using backpropagation. Additionally, the processor (130) can perform inference for a specific purpose using a model implemented in an artificial neural network manner.

[0118] Hereinafter, we will look at FIG. 10. The external correction method illustrated in FIG. 10 can be performed by an external correction device (100) including a processor (130).

[0119] Meanwhile, FIG. 10 is merely illustrative, and the concept of the present invention is not to be interpreted as being limited to that illustrated in FIG. 10. For example, each step may be configured in a different order than that illustrated in FIG. 10, at least one of the steps illustrated in FIG. 10 may not be performed, or one or more steps not illustrated in FIG. 10 may be additionally performed. Below, the external correction method based on semantic partitioning will be described in order. Since the operation (function) of the external correction method according to the embodiment is essentially the same as the function of the system, descriptions that overlap with FIG. 2 to FIG. 9 will be omitted.

[0120] FIG. 10 is a diagram showing an external correction process according to an embodiment.

[0121] In step S110, camera and lidar data are collected. In step S120, external calibration is performed. In the embodiment, during the first external calibration process, the difference (Dt) between the camera data collection time and the lidar data collection time is assumed to be 0. This is because the difference between data collection times is generally smallest during the first external calibration. In step S130, the difference between data collection times is estimated. In the embodiment, in step S130, external parameters can be detected and the difference between data collection times can be estimated. In step S140, the suitability of the external parameters is determined, and if the suitability is above a certain level, the corresponding external parameters are used. If the suitability of the external parameters is below a certain level in step S140, the process returns to step S120.

[0122] The semantic segmentation-based external correction device and method according to the embodiment can save costs and time by automating the correction process through the use of semantic segmentation to automatically perform correction between the camera and the lidar.

[0123] In addition, through the embodiment, the same object (building, vehicle, person, etc.) is recognized and aligned in the camera and lidar, so that the data of the two sensors is precisely matched, thereby increasing the accuracy of data alignment between sensors.

[0124] In addition, through the embodiments, sensor fusion performance can be improved in autonomous vehicles, robot vision, and AR / VR systems.

[0125] Furthermore, while conventional methods required placing artificial markers such as checkerboards or specific patterns in the environment, the embodiment provides a more practical sensor calibration method by performing calibration using semantic segmentation in a real environment (natural environment). Additionally, the calibration method provided in the embodiment can be utilized in various environments requiring sensor fusion, such as autonomous driving, drones, and robots, and enables calibration in multiple environments including roads, indoor environments, and factory automation. Moreover, the embodiment improves the object recognition accuracy of autonomous vehicles and the path planning reliability of robots by aligning sensor data more precisely using semantic segmentation through machine learning. Furthermore, the embodiment enables the improvement of AI model performance by enhancing the reliability of AI training data through the provision of accurate sensor data.

[0126] Meanwhile, the methods according to the various embodiments of the present invention described above can be implemented in the form of an application or software program that can be installed on an existing electronic device.

[0127] In addition, the whole or part of the method may be composed of multiple software function modules and implemented on an operating system (OS). Alternatively, each step may be composed of a single software function module, or each step may be combined to form a single software function module and implemented on an operating system. Therefore, even if all of the embodiments of the present disclosure are not implemented as a single software function module, if multiple software function modules implement each step of the present disclosure and multiple software function modules are implemented on a single operating system, it can be understood that the method of the present disclosure has been implemented.

[0128] In addition, the methods according to the various embodiments of the present invention described above can be implemented solely through software upgrades or hardware upgrades of existing electronic devices. Furthermore, the various embodiments of the present invention described above can also be performed through an embedded server equipped in an electronic device or an external server of the electronic device.

[0129] Meanwhile, according to one embodiment of the present invention, the various embodiments described above may be implemented as software comprising instructions stored on a computer-readable recording medium using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented as the processor itself. According to the software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

[0130] Meanwhile, a computer or a similar device may include a device according to the disclosed embodiments, which is capable of calling instructions stored from a storage medium and operating according to the called instructions. When said instructions are executed by a processor, the processor may perform a function corresponding to said instructions directly or by using other components under the control of said processor. The instructions may include code generated or executed by a compiler or an interpreter.

[0131] A computer-readable recording medium may be provided in the form of a non-transitory computer-readable recording medium. Here, "non-transitory" simply means that the storage medium does not contain a signal and is tangible, without distinguishing whether data is stored semi-permanently or temporarily on the storage medium. In this context, a non-transitory computer-readable medium refers to a medium that stores data semi-permanently and is readable by a device, rather than a medium that stores data for a short moment, such as registers, caches, or memory. Specific examples of non-transitory computer-readable media may include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.

[0132] As described above, exemplary embodiments have been disclosed in the drawings and specification. Although specific terms have been used to describe the embodiments in this specification, they are used only for the purpose of explaining the technical concept of this disclosure and are not intended to limit the meaning or the scope of this disclosure as defined in the claims. Therefore, those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of this disclosure should be determined by the technical concept of the appended claims.

Claims

Claim 1 An external correction device based on semantic segmentation, comprising: a communication module; a memory storing at least one instruction; and a processor, wherein the at least one instruction is executed by the processor, the external correction device collects image data from a camera and collects point cloud data from a lidar, performs semantic segmentation based on the collected image and point cloud data, aligns the semantic segmentation results obtained from the camera and lidar to detect identical objects, extracts extrinsic parameters for determining the relative position and orientation between the camera and lidar using the detection results, transforms the data collected from the camera and lidar to have the same coordinate system based on the extrinsic parameters, and the extrinsic parameters include a rotation matrix (R) and a translation vector (t) of the camera and lidar. Claim 2 In claim 1, the external correction device evaluates fitness during the external correction process between the camera and the LiDAR, wherein the fitness is the accuracy of the estimated extrinsic parameter. Claim 3 In paragraph 2, the external correction device measures the alignment accuracy of the LiDAR point cloud on the camera image to calculate the suitability, and the suitability is expressed in mathematical formula 1 An external correction device calculated through, wherein in the above mathematical formula 1, f1(x) is the number of LiDAR points existing in the camera image coordinate system, f2(x) is the number of times the semantic information of the LiDAR and the camera matches, and α and β in the above mathematical formula 1 are weights. Claim 4 In paragraph 3, the external correction device uses f1(x) of Equation 1 as Equation 2 It is calculated through, and the above f2(x) is mathematical formula 3 An external correction device calculated through Claim 5 In claim 1, the external correction parameter includes a rotation parameter, the rotation parameter includes roll rotation (rot_x) and yaw rotation (rot_z) values, and the external correction device detects an optimal peak at a specific rotation value. Claim 6 In claim 5, the external correction device calculates the highest fitness point as the optimal solution through a nonlinear search function including a Genetic Algorithm (GA) and Article Swarm Optimization (PSO). Claim 7 In claim 1, the external correction device performs time synchronization by comparing the time stamps of the camera and the lidar to match the closest data.

Images