Radar-camera sensor fusion-based intelligent object tracking system system and method thereof

The radar-camera sensor fusion system addresses sensor data inconsistencies and computational inefficiencies by synchronizing and tracking objects, improving accuracy and predicting collision risks for enhanced safety.

KR103046526B1Active Publication Date: 2026-11-10HUMAN DEFENCE
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
KR1020260117142
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2026-06-26
Publication Date
2026-11-10
Estimated Expiration
2046-06-26

AI Technical Summary

Technical Problem

Existing object recognition technologies face challenges in accurately tracking and predicting the movement of objects in diverse environments due to inconsistencies in sensor data coordination, computational inefficiencies, and the inability to predict collision risks, leading to reduced reliability and increased false alarms.

Method used

A radar-camera sensor fusion system that synchronizes camera and radar data spatiotemporally on a common reference point and coordinate system, extracts regions of interest, and tracks object movement to predict collision risks and output warnings.

Benefits of technology

Enhances object recognition accuracy, reduces computational load, and stabilizes tracking of multiple objects, enabling timely collision risk prediction and prevention.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112026077998787-PAT00004_ABST
    Figure 112026077998787-PAT00004_ABST
Patent Text Reader

Abstract

An intelligent object tracking system and method based on radar-camera sensor fusion are disclosed. An intelligent object tracking system based on radar-camera sensor fusion according to an embodiment of the present invention comprises: an image data collection unit that collects real-time image data from at least one camera terminal; a radar data collection unit that collects radar data including at least one of distance information, azimuth information, and velocity information from at least one radar sensor; a spatiotemporal synchronization unit that generates synchronization data by aligning the real-time image data collected through the image data collection unit and the radar data collected through the radar data collection unit onto a preset identical reference point and a preset identical reference coordinate system; an interest region setting unit that extracts at least one object candidate region from the real-time image data based on the synchronization data aligned by the spatiotemporal synchronization unit and sets an interest region corresponding to the object candidate region; and an object tracking unit that classifies a target object within the interest region set by the interest region setting unit and tracks the movement state of the target object. and includes an object risk analysis unit that analyzes the movement trajectory of the target object classified by the object tracking unit to calculate the possibility of collision risk of the target object, and outputs a warning signal when the possibility of collision risk is greater than or equal to a preset threshold.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to the field of object recognition and tracking technology, and more specifically, to a radar-camera sensor fusion-based intelligent object tracking system and method capable of setting a region of interest, classifying target objects, tracking movement status, and analyzing collision risk by synchronizing and fusing image data collected from a camera with distance information, azimuth information, and speed information collected from a radar sensor in space and time. Background Technology

[0002] Recently, there has been an increasing demand for intelligent monitoring and warning technologies to prevent safety accidents at intersections, crosswalks, roadsides, and various mobility environments by recognizing diverse objects—such as pedestrians, vehicles, wheelchairs, and electric mobility devices—in real time and tracking their movement status. In particular, technology that provides warnings by preemptively determining whether a pedestrian is attempting to cross or the likelihood of a collision with a vehicle is recognized as a crucial element in improving traffic safety.

[0003] Traditionally, camera-based image recognition technology has been widely used. Cameras offer the advantage of being suitable for object classification because they can capture a relatively rich range of object features, including their external shape, contours, color, and morphological characteristics. However, camera-based methods suffer from a problem where image quality degrades in low-light or adverse weather conditions, such as night, backlight, fog, rain, or snow, which can lead to a sharp decline in object detection and classification accuracy. Furthermore, performing object detection and classification operations across the entire image area can result in an excessive increase in computational load, making it difficult to secure real-time processing performance.

[0004] Meanwhile, object detection technology using radar sensors can relatively reliably acquire physical quantities such as distance, azimuth, and speed, making it advantageous for identifying the presence and movement of objects even at night or in adverse weather conditions. However, radar-based methods have limitations in precisely distinguishing the shape or specific type of an object, which can make it difficult to accurately classify pedestrians, wheelchair users, vehicles, and other moving objects. Furthermore, it may be difficult to precisely interpret an object's detailed behavior or entry intent based solely on radar detection results.

[0005] Furthermore, even when using cameras and radar together, the data generated by the two sensors differs in coordinate systems and formats, as well as sampling periods and update times; therefore, it is difficult to accurately match identical objects using only simple parallel collection. Such spatiotemporal inconsistencies can lead to matching errors between sensors, resulting in distorted object locations or reduced reliability in determining whether an object is identical. Consequently, it is necessary to ensure consistency through spatial and temporal alignment between the differing sensor data.

[0006] In conventional technology, object recognition algorithms are often performed on the entire camera image. This results in repetitive computations even in areas where actual objects do not exist, leading to reduced processing speed and a failure to sufficiently ensure the real-time performance required in edge environments. In particular, in environments where multiple objects appear simultaneously, redundant computations increase further, which can lead to greater inefficiency in system resources.

[0007] Furthermore, conventional object recognition technologies often remain at the level of simply determining the presence of an object or tracking its current location, lacking the ability to predict future entry intentions or collision possibilities based on the object's movement trajectory. However, in actual safety warning systems, it is more important to predict the future movement path of an object and to calculate the estimated collision time by analyzing the relative positional and relative velocity relationships between multiple objects, rather than simply detecting objects. If these functions are not sufficiently implemented, there is a problem that warnings cannot be provided at the appropriate time or that unnecessary false alarms may increase.

[0008] Therefore, a new radar-camera sensor fusion-based intelligent object tracking system and method are required that can spatiotemporally synchronize camera data and radar data having different characteristics at the same reference point and the same reference coordinate system, improve computational efficiency through radar-based region of interest setting, track target objects by fusing image feature data and radar data, and output a warning signal by analyzing the entry intention and collision risk of the target objects. Prior art literature

[0009] Korean Registered Patent Publication No. 10-1569411 Korean Registered Patent Publication No. 10-2290353 The problem to be solved

[0010] The present invention aims to provide an intelligent object tracking system and method based on radar-camera sensor fusion that can fuse data obtained from different sensors in a consistent manner by synchronizing real-time image data collected from a camera terminal and radar data collected from a radar sensor spatiotemporally at the same reference point and the same reference coordinate system.

[0011] In addition, the purpose is to provide an intelligent object tracking system and method based on radar-camera sensor fusion that can improve the efficiency of object recognition while reducing unnecessary computations by extracting object candidate regions from real-time image data based on spatiotemporally aligned synchronization data and setting corresponding regions of interest.

[0012] In addition, the purpose is to provide an intelligent object tracking system and method based on radar-camera sensor fusion that can more accurately recognize target objects in various environments by combining image feature data and radar data within the region of interest to determine whether they are the same object and classifying the type of target object.

[0013] In addition, the purpose is to provide a radar-camera sensor fusion-based intelligent object tracking system and method that can stably track the continuous movement state of multiple target objects by tracking the position change and movement direction of the target objects based on the object information and temporal change of the movement trajectory of the classified target objects.

[0014] In addition, the purpose is to provide an intelligent object tracking system and method based on radar-camera sensor fusion that can predict the entry intention of a target object and calculate the expected collision time based on whether the predicted movement paths between target objects overlap, the relative position relationship, and the relative speed relationship, thereby analyzing the collision risk and outputting a warning signal corresponding to the risk level to induce the prevention of dangerous accidents.

[0015] The problems that the present invention aims to solve are not limited to those mentioned above, and other problems that the present invention aims to solve that are not mentioned herein will be clearly understood by those skilled in the art to which the present invention belongs from the description below. means of solving the problem

[0016] An intelligent object tracking system based on radar-camera sensor fusion according to an embodiment of the present invention comprises: an image data collection unit that collects real-time image data from at least one camera terminal; a radar data collection unit that collects radar data including at least one of distance information, azimuth information, and speed information from at least one radar sensor; a spatiotemporal synchronization unit that generates synchronization data by aligning real-time image data collected through the image data collection unit and radar data collected through the radar data collection unit onto a preset identical reference time and a preset identical reference coordinate system; an interest area setting unit that extracts at least one object candidate area from real-time image data based on the synchronization data aligned by the spatiotemporal synchronization unit and sets an interest area corresponding to the object candidate area; an object tracking unit that classifies a target object within the interest area set by the interest area setting unit and tracks the movement state of the target object; and an object risk analysis unit that calculates the possibility of a collision risk of the target object by analyzing the movement trajectory of the target object classified by the object tracking unit and outputs a warning signal when the possibility of a collision risk is greater than or equal to a preset threshold.

[0017] In addition, the spatiotemporal synchronization unit is characterized by including a spatial alignment unit that calculates a transformation matrix and performs spatial alignment based on the correspondence relationship between 2D image coordinates extracted from real-time video data and 3D spatial coordinates extracted from radar data, and a time alignment unit that corrects the difference between the exposure time extracted from real-time video data and the scan time extracted from radar data.

[0018] In addition, the interest region setting unit is characterized by including an object candidate extraction unit that extracts at least one object candidate region from real-time image data and generates image feature data corresponding to the object candidate region, an interest region variable setting unit that determines the size of the interest region based on the size of the object candidate region, a projection mapping unit that calculates the center coordinates of the interest region by projecting radar data onto the planar coordinates of the real-time image data, and an operation optimization unit that groups the object candidate regions into one object unit interest region when the distance between the object candidate regions is less than a preset reference distance.

[0019] In addition, the object tracking unit is characterized by including an identical object determination unit that determines whether an object is the same and generates object information based on image feature data and radar data corresponding to object candidate regions included within the region of interest, an object classification unit that classifies the type of target object based on the object information generated through the identical object determination unit, and a movement state tracking unit that tracks the position change and movement direction of the target object based on the temporal change of the object information.

[0020] Additionally, the object risk analysis unit is characterized by including: a behavior prediction unit that predicts the entry intention of a target object based on the movement trajectory of a target object tracked through an object tracking unit; a collision prediction unit that calculates the predicted movement path of a target object and determines whether the predicted movement paths between target objects overlap; a collision time calculation unit that calculates the expected collision time between target objects based on at least one of the relative positional relationship of the movement paths between target objects, the relative positional relationship of the predicted movement paths between target objects, and the relative speed relationship between target objects when there are multiple target objects whose predicted movement paths overlap in the collision prediction unit; and a warning control unit that classifies a risk level according to the expected collision time and the type of target object and outputs a warning signal corresponding to the risk level. Effects of the invention

[0021] The radar-camera sensor fusion-based intelligent object tracking system and method of the present invention have the effect of fusion of data obtained from different sensors in a consistent manner by synchronizing real-time image data collected from a camera terminal and radar data collected from a radar sensor spatiotemporally on the same reference point and the same reference coordinate system.

[0022] In addition, by extracting object candidate regions from real-time image data based on spatiotemporally aligned synchronization data and setting corresponding regions of interest, it has the effect of improving the efficiency of object recognition while reducing unnecessary computations.

[0023] In addition, by combining image feature data and radar data within the region of interest to determine whether they are the same object and classifying the type of target object, it has the effect of recognizing target objects more accurately in various environments.

[0024] In addition, by tracking the position change and direction of movement of a target object based on the object information and temporal change of the movement trajectory of the classified target object, it has the effect of stably tracking the continuous movement state of multiple target objects.

[0025] In addition, by predicting the entry intention of target objects and calculating the estimated collision time based on whether predicted movement paths overlap between target objects, relative positional relationships, and relative speed relationships, it analyzes the collision risk and outputs a warning signal corresponding to the risk level, thereby having the effect of preventing dangerous accidents in advance. Brief explanation of the drawing

[0026] FIG. 1 is a configuration diagram of an intelligent object tracking system based on radar-camera sensor fusion according to an embodiment of the present invention. FIG. 2 is a diagram illustrating a spatiotemporal synchronization unit of an intelligent object tracking system based on radar-camera sensor fusion according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a region of interest setting unit of an intelligent object tracking system based on radar-camera sensor fusion according to an embodiment of the present invention. FIG. 4 is a diagram illustrating an object tracking unit of a radar-camera sensor fusion-based intelligent object tracking system according to an embodiment of the present invention. FIG. 5 is a diagram illustrating an object risk analysis unit of a radar-camera sensor fusion-based intelligent object tracking system according to an embodiment of the present invention. Specific details for implementing the invention

[0027] Specific details regarding the problem to be solved, the means for solving the problem, and the effects of the invention as described above are included in the embodiments and drawings to be described below. The advantages and features of the present invention, and the methods for achieving them, will become clear by referring to the embodiments described below in detail together with the accompanying drawings.

[0029] Hereinafter, the present invention will be described in more detail with reference to the attached drawings.

[0030] FIG. 1 is a configuration diagram of an intelligent object tracking system based on radar-camera sensor fusion according to an embodiment of the present invention, FIG. 2 is a diagram for explaining a spatiotemporal synchronization unit of an intelligent object tracking system based on radar-camera sensor fusion according to an embodiment of the present invention, FIG. 3 is a diagram for explaining a region of interest setting unit of an intelligent object tracking system based on radar-camera sensor fusion according to an embodiment of the present invention, FIG. 4 is a diagram for explaining an object tracking unit of an intelligent object tracking system based on radar-camera sensor fusion according to an embodiment of the present invention, and FIG. 5 is a diagram for explaining an object risk analysis unit of an intelligent object tracking system based on radar-camera sensor fusion according to an embodiment of the present invention.

[0032] Referring to FIG. 1, a radar-camera sensor fusion-based intelligent object tracking system (100) according to an embodiment of the present invention may include an image data collection unit (110), a radar data collection unit (120), a spatiotemporal synchronization unit (130), a region of interest setting unit (140), an object tracking unit (150), and an object risk analysis unit (160).

[0033] The image data collection unit (110) collects real-time image data from at least one camera terminal, and the radar data collection unit (120) collects radar data including at least one of distance information, azimuth information, and speed information from at least one radar sensor, and the spatiotemporal synchronization unit (130) generates synchronization data by aligning the real-time image data collected through the image data collection unit (110) and the radar data collected through the radar data collection unit (120) with a preset identical reference point and a preset identical reference coordinate system, and the interest area setting unit (140) extracts at least one object candidate area from the real-time image data based on the synchronization data aligned by the spatiotemporal synchronization unit (130), and sets an interest area corresponding to the object candidate area, and the object tracking unit (150) classifies target objects within the interest area set by the interest area setting unit (140) and tracks the movement state of the target objects, and the object risk analysis unit (160) classifies the target objects classified by the object tracking unit (150). The movement trajectory of an object is analyzed to calculate the likelihood of a collision risk of the target object, and if the likelihood of a collision risk is greater than or equal to a preset threshold, a warning signal can be output.

[0035] More specifically, the image data collection unit (110) may include at least one camera terminal installed in a fixed or mobile form to continuously collect real-time image data of a surveillance target area. At this time, the camera terminal may be provided as a visible light camera, a low-light camera, an infrared camera, or a combination thereof, and may be deployed in response to various surveillance environments such as crosswalks, intersections, roadsides, parking areas, or pedestrian traffic areas.

[0036] In addition, the image data collection unit (110) can store the collected real-time image data together with shooting time information, frame number, exposure time information and camera internal parameters.

[0038] For example, the image data collection unit (110) can perform at least one of resolution adjustment, frame alignment, distortion correction, noise removal, and contrast correction on the collected real-time image data.

[0039] In addition, the image data collection unit (110) may also generate frame-unit image feature data to be advantageous for detecting object candidate regions.

[0041] The radar data collection unit (120) can collect radar data from at least one radar sensor, including millimeter wave radar, FMCW radar, etc. The radar data may include at least one of distance information, azimuth information, and velocity information of a target object, and may further include reflection intensity information, Doppler information, cluster center information, or point cloud information as needed.

[0042] Accordingly, the radar data collection unit (120) can collect physical movement characteristics of a target object that are difficult to obtain with only the image data collection unit (110).

[0043] For example, the radar data collection unit (120) can enable temporal alignment with the real-time image data by storing scan time information and timestamp information corresponding to the radar data together.

[0045] The above-mentioned spatiotemporal synchronization unit (130) can calculate a transformation matrix based on the correspondence between 2D image coordinates extracted from the real-time image data and 3D spatial coordinates extracted from the radar data, and perform spatial alignment using the transformation matrix.

[0046] More specifically, as illustrated in FIG. 2, the spatiotemporal synchronization unit (130) may include a spatial alignment unit (131) that calculates a transformation matrix and performs spatial alignment based on the correspondence between 2D image coordinates extracted from the real-time image data and 3D spatial coordinates extracted from the radar data, and a time alignment unit (132) that corrects the difference between the exposure time extracted from the real-time image data and the scan time extracted from the radar data.

[0047] The spatial alignment unit (131) can establish a mutual correspondence relationship between the image coordinate system of the camera terminal and the spatial coordinate system of the radar sensor, and based on this, align the real-time image data and the radar data on the same reference coordinate system.

[0048] At this time, the spatial alignment unit (131) can calculate a transformation matrix by matching the 2D image coordinates of a reference point detected from the real-time image data with the 3D spatial coordinates of a corresponding reference point obtained from the radar data. Accordingly, the spatial alignment unit (131) can reduce coordinate mismatch between heterogeneous data obtained from different sensors and improve the accuracy of data correspondence for the same object.

[0050] Additionally, the spatial alignment unit (131) can repeatedly correct the transformation matrix based on a plurality of corresponding reference points obtained during the calibration process. For example, the spatial alignment unit (131) can reduce spatial alignment errors that may occur due to sensor installation errors, parallax, or environmental changes by calculating the error between image coordinates and radar coordinates for a preset reference object or reference pattern and updating the transformation matrix to minimize the error.

[0052] For example, the spatial alignment unit (131) can use the transformation matrix to transform the position of an object included in the radar data into planar coordinates of the real-time image data and correspond the transformed coordinates to an object candidate region or region of interest on the real-time image data.

[0054] At this time, the transformation matrix calculated by the spatial alignment unit (131) is information for mathematically expressing the geometric correspondence between the 2D image coordinate system of the camera terminal and the 3D spatial coordinate system of the radar sensor. The transformation matrix can be used to correspond the spatial position of a target object included in the radar data to a specific pixel position or specific area on the real-time image data, and can be utilized as reference information to correct coordinate system mismatches caused by differences in installation positions, installation angles, and reference directions between sensors.

[0055] That is, the above transformation matrix can function as a coordinate transformation standard for determining which location in the camera image corresponds to an object detected by the radar sensor.

[0057] In addition, the transformation matrix is ​​not limited to information that simply substitutes coordinates, but may include parameter information for converting an object position corresponding to distance information, azimuth information, or velocity information of the radar data into interpretable position information of the real-time image data by reflecting the relative positional and attitude relationships between the camera terminal and the radar sensor.

[0058] For example, the transformation matrix may include coordinate transformation relationships derived from a plurality of corresponding points, and the spatial alignment unit (131) may use the transformation matrix to map one point or a plurality of points on the radar coordinate system to one point or area on the image plane.

[0059] Therefore, the more accurately the transformation matrix is ​​calculated, the better the correspondence accuracy between the object location detected by the radar sensor and the object candidate region in the camera image can be improved, and as a result, the reliability of sensor fusion-based object recognition and tracking can be increased.

[0061] Meanwhile, the time alignment unit (132) can correct the difference between the exposure time extracted from the real-time image data and the scan time extracted from the radar data, thereby aligning the real-time image data and the radar data so that they correspond to information at substantially the same time.

[0062] Generally, since camera terminals and radar sensors generate data at different intervals, position errors or movement trajectory errors may occur due to differences in the collection time, even if the information is about the same object. Therefore, the time alignment unit (132) can improve the temporal alignment of sensor fusion by correcting these differences in time.

[0063] For example, the time alignment unit (132) can set a reference point by comparing the timestamps assigned to the real-time image data and the radar data, respectively, and correct the difference between the exposure point and the scan point to correspond to the reference point.

[0064] Additionally, when the radar data is collected at a lower frequency than the real-time image data, the time alignment unit (132) can generate virtual radar data at an intermediate time point based on radar data between two adjacent scan timestamps to generate radar information corresponding to the camera frame time point.

[0066] In addition, the time alignment unit (132) can estimate the intermediate position of the target object using the previous position, current position, and movement speed information of the target object.

[0067] For example, the time alignment unit (132) can calculate radar position information of the target object corresponding to the frame acquisition time of the camera terminal through at least one of linear interpolation, curve interpolation, and prediction model-based correction.

[0068] Thus, the time alignment unit (132) can mitigate object position discrepancies that may occur due to differences in update cycles per sensor.

[0070] For example, the space-time synchronization unit (130) generates the synchronization data by combining the spatial alignment result by the spatial alignment unit (131) and the temporal alignment result by the temporal alignment unit (132), wherein the synchronization data may include image feature data and radar object information corresponding to the same reference coordinate system and the same reference time.

[0071] That is, the space-time synchronization unit (130) can support reliable fusion of data acquired from different sensors by securing coordinate alignment between the real-time image data and the radar data through the space alignment unit (131) and securing time alignment between the real-time image data and the radar data through the time alignment unit (132).

[0073] The above-mentioned region of interest setting unit (140) can extract at least one object candidate region from the real-time image data based on the synchronization data generated by the above-mentioned spatiotemporal synchronization unit (130), and generate image feature data corresponding to the object candidate region. At this time, the object candidate region can be detected based on the object's outline, edge, texture, brightness distribution, contour, or preset deep learning features. Additionally, the above-mentioned region of interest setting unit (140) can variably determine the range of the region of interest according to the size or location occupied by the object candidate region within the image.

[0074] Referring to FIG. 3, the interest region setting unit (140) may include an object candidate extraction unit (141) that extracts at least one object candidate region from the real-time image data and generates image feature data corresponding to the object candidate region, an interest region variable setting unit (142) that determines the size of the interest region based on the size of the object candidate region, a projection mapping unit (143) that calculates the center coordinates of the interest region by projecting the radar data onto the planar coordinates of the real-time image data, and a computation optimization unit (144) that groups the object candidate regions into one object unit interest region when the distance between the object candidate regions is less than a preset reference distance.

[0075] More specifically, the object candidate extraction unit (141) can detect an area presumed to be an object within the real-time image data, extract at least one object candidate area, and generate image feature data corresponding to each object candidate area.

[0076] For example, the object candidate extraction unit (141) can distinguish between an object and a background based on at least one of the brightness change of pixels in an image, edge components, contour information, texture pattern, color distribution and feature map.

[0077] At this time, the object candidate extraction unit (141) can generate feature vectors and feature maps for each object candidate region using a pre-configured deep learning-based feature extraction model.

[0079] For example, the object candidate extraction unit (141) can prioritize searching for an image area with a high probability of object existence by referring to radar-based location information included in the synchronization data provided by the spatiotemporal synchronization unit (130).

[0080] Accordingly, the object candidate extraction unit (141) can reduce unnecessary detection operations by extracting object candidate regions centered on the region corresponding to the location detected by the radar sensor, instead of uniformly searching the entire image.

[0081] In addition, the object candidate extraction unit (141) can generate refined object candidate regions by separating or merging candidate regions according to preset criteria when multiple candidate regions overlap for a single object or when adjacent objects are excessively merged into a single region.

[0083] Meanwhile, the variable setting unit (142) of the interest region can determine the range of the interest region based on at least one of the size, location, aspect ratio, and the ratio that the object candidate region occupies within the image.

[0084] For example, the variable setting unit (142) of the interest region may asymmetrically adjust the width or height of the interest region by considering the movable direction of the object candidate region, the degree of separation from the image edge, and the possibility of expansion in subsequent frames.

[0085] Accordingly, the variable setting unit (142) of the interest area can absorb object position changes or radar coordinate projection errors that may occur in subsequent frames by setting an area of ​​interest that is slightly expanded compared to the actual range occupied by the target object within the image.

[0086] That is, the variable setting unit (142) of the interest area can improve the continuity and stability of object tracking by dynamically adjusting the interest area according to the state of the object candidate area, rather than setting the interest area to a fixed size.

[0088] The projection mapping unit (143) can project the radar data onto the planar coordinates of the real-time image data to calculate the center coordinates of the region of interest.

[0089] More specifically, the projection mapping unit (143) can convert distance information and azimuth information of an object detected in a radar coordinate system into coordinates on an image plane based on at least one of the transformation matrix and coordinate correspondence information calculated by the spatiotemporal synchronization unit (130). Accordingly, the projection mapping unit (143) can correspond the object location detected first by the radar sensor to a specific point on the image and set the center of the region of interest based on said point.

[0090] For example, the projection mapping unit (143) can compare the center coordinates of the object candidate area detected by the object candidate extraction unit (141) with the coordinates projected from the radar data, and if the difference between the two is within a preset allowable range, determine that this is information corresponding to the same object.

[0091] On the other hand, if the difference exceeds a preset allowable range, it is determined that there is a possibility of projection error, noise, or false detection, and the priority of setting the region of interest for that area may be adjusted or a re-detection procedure may be performed.

[0092] In this way, the projection mapping unit (143) can increase the reliability of the region of interest center by combining image-based candidate regions and radar-based location information.

[0094] The above operation optimization unit (144) can group the object candidate regions into one object unit interest region when the distance between the object candidate regions is less than a preset reference distance.

[0095] For example, the operation optimization unit (144) can group multiple object candidate regions into a single group and set them as a single region of interest when multiple object candidate regions are arranged adjacently to each other or when multiple detailed candidate regions are generated for a single actual object. Accordingly, the operation optimization unit (144) can prevent duplicate operations on the same object and reduce repetitive inference for multiple detailed regions.

[0096] Additionally, the operation optimization unit (144) can control subsequent object classification or tracking operations to be performed only within the region of interest. That is, the operation optimization unit (144) can omit or reduce subsequent analysis for image regions where the probability of an object existing is low, and can designate the region of interest set through the object candidate extraction unit (141), the variable setting unit of the region of interest (142), and the projection mapping unit (143) as a priority processing target.

[0097] Accordingly, the above-mentioned interest area setting unit (140) can reduce the amount of computation compared to a full processing method for the entire image, while performing more intensive analysis on areas where there is a high probability that actual objects exist.

[0099] That is, the interest region setting unit (140) can generate object candidate region and image feature data through the object candidate extraction unit (141), dynamically set the range of the interest region corresponding to the state of the object candidate region through the interest region variable setting unit (142), reflect the radar-based object location as the center of the interest region on the image plane through the projection mapping unit (143), and improve computation efficiency by grouping adjacent object candidate regions through the computation optimization unit (144).

[0101] Meanwhile, the object tracking unit (150) can determine whether the object is the same and generate object information based on image feature data corresponding to the object candidate region included within the area of ​​interest and the radar data.

[0102] More specifically, the object tracking unit (150) can determine whether the object is the same by correlating the external features, contour features, or classification reliability included in the image feature data with the distance information, azimuth information, velocity information, or reflection characteristics included in the radar data. Through this, it is possible to reliably determine whether the image-based object and the radar-based object correspond to the same real-world object.

[0103] In addition, the object tracking unit (150) can classify the type of the target object based on the object information. For example, the object tracking unit (150) can classify the target object into at least one of a pedestrian, a vehicle, a wheelchair user, a two-wheeled vehicle, or other vehicles, and the result of such object type classification can be used for tracking movement status and risk analysis. Furthermore, the object tracking unit (150) can continuously track the location change, direction of movement, speed of movement, and trajectory information of the target object based on the temporal change of the object information.

[0105] For example, as illustrated in FIG. 4, the object tracking unit (150) may include a same object determination unit (151) that determines whether the object is the same and generates object information based on image feature data corresponding to the object candidate area included within the area of ​​interest and the radar data, an object classification unit (152) that classifies the type of the target object based on the object information generated through the same object determination unit (151), and a movement state tracking unit (153) that tracks the position change and movement direction of the target object based on the temporal change of the object information.

[0106] More specifically, the same object determination unit (151) can analyze the correspondence between image feature data corresponding to the object candidate region included within the region of interest and the radar data to determine whether information obtained from different sensors represents the same real-world object, and generate object information based on the result of the determination.

[0107] For example, the same object determination unit (151) can compare the center coordinates, size, contour, external features, and deep learning-based feature vector of the object candidate area with the distance information, azimuth information, speed information, reflection intensity information, and cluster center information included in the radar data to calculate the degree of alignment between them.

[0108] Accordingly, the same object determination unit (151) can more reliably determine whether the image-based detection result and the radar-based detection result correspond to the same target object.

[0109] For example, the same object determination unit (151) can determine whether the image feature data and the radar data are the same object by using coordinate alignment information and viewpoint alignment information included in the synchronization data generated by the spatiotemporal synchronization unit (130), so that the image feature data and the radar data are aligned to correspond to the same reference coordinate system and the same reference viewpoint.

[0110] At this time, the same object determination unit (151) can determine whether it is the same object by using at least one of the difference between the center coordinates of the object candidate area and the coordinates of the radar data projected onto the image plane, the relationship between the size of the object candidate area and the object size estimated by the radar data, and the alignment between the direction of movement of the object candidate area and the velocity vector of the radar data.

[0111] In addition, the same object determination unit (151) can generate object information including object identification information, location information, speed information, direction of travel information, image feature information and radar feature information based on the result of determining whether the objects are the same.

[0112] For example, the object information may be provided in a form that integrates image-based feature information and radar-based physical information for a single target object.

[0114] Meanwhile, the object classification unit (152) can classify the type of the target object based on the object information generated through the identical object determination unit (141). More specifically, the object classification unit (152) can perform classification that simultaneously reflects the external characteristics and physical movement characteristics of the target object by using image feature information and radar feature information included in the object information together. Accordingly, the object classification unit (152) can determine the type of the target object with higher classification accuracy compared to a camera-only based method or a radar-only based method.

[0115] For example, the object classification unit (152) can classify the target object into at least one of a pedestrian, a vehicle, a bicycle, an electric scooter, a wheelchair user, and other moving objects. At this time, the object classification unit (152) can derive a classification result by comprehensively considering the outer shape, posture, size, contour, or brightness distribution of the object included in the image feature data, and the distance change, relative speed, azimuth change, reflection characteristics included in the radar data.

[0116] For example, even if they have similar appearances in the image, if the radar-based speed pattern or reflection characteristics are different, the object classification unit (152) can use this as a distinguishing factor to perform more detailed object classification.

[0117] In addition, the object classification unit (152) can set not only the type of the target object but also the risk priority and protection priority of the target object.

[0118] For example, if an object is classified as a vulnerable road user, such as a pedestrian, child, the elderly, or a wheelchair user, classification information may be provided so that it can be used as a target for attention.

[0120] Meanwhile, the movement state tracking unit (153) can track the location change and movement direction of the target object based on the temporal change of the object information. More specifically, the movement state tracking unit (153) can calculate the current location, previous location, movement direction, movement speed, and movement trajectory of the target object by comparing object information generated at consecutive points in time.

[0121] Accordingly, the movement state tracking unit (153) can continuously track how the target object moves over time, going beyond static object detection results in a single frame unit.

[0123] For example, the movement state tracking unit (153) can generate a movement path of the target object based on at least one of the amount of change in object position between frames, the amount of change in radar-based speed, the movement direction vector, and a preset tracking model.

[0124] In addition, the movement state tracking unit (153) can calculate the estimated position of the target object using the position information and radar-based speed information from the previous point in time, even when the object is temporarily obscured or image features are weakened.

[0125] In addition, the movement state tracking unit (153) may track individual movement states based on object identification information for each target object even when multiple target objects exist simultaneously.

[0126] For example, the movement state tracking unit (153) may assign a unique identifier to each target object and manage the same identifier to be maintained based on the consistency of object information at each point in time. That is, the movement state tracking unit (153) can track the movement trajectory of each target object individually while reducing confusion between objects, even when multiple objects cross or move adjacent to each other.

[0128] The object risk analysis unit (160) can predict the entry intention of the target object based on the movement trajectory of the target object tracked through the object tracking unit (150). More specifically, the object risk analysis unit (160) can determine whether the target object intends to enter a specific area based on the target object's current location, direction of travel, change in stride length, change in speed, or change in acceleration. For example, if a pedestrian approaches the boundary of a crosswalk and the direction of travel is formed toward the roadway, the object risk analysis unit (160) can determine this as a state with a high probability of entry.

[0129] Additionally, the object risk analysis unit (160) can calculate the predicted movement path of the target object and perform collision prediction by determining whether the predicted movement paths overlap between the target objects. If there are multiple target objects whose predicted movement paths overlap, the object risk analysis unit (160) can calculate the estimated collision time between the target objects based on at least one of the relative positional relationship of the movement paths between the target objects, the relative positional relationship of the predicted movement paths between the target objects, and the relative speed relationship between the target objects. Accordingly, the object risk analysis unit (160) can selectively detect situations where the probability of a future collision is actually high, rather than simply generating a warning based only on the current location.

[0130] For example, the object risk analysis unit (160) can classify risk levels according to the expected collision time and the type of each target object. For example, the shorter the expected collision time, the higher the risk level can be classified, and if objects with a high priority for protection, such as vulnerable road users or pedestrians, are included, it can be determined to be a higher risk level even if the expected collision time is the same.

[0131] In addition, the object risk analysis unit (160) can output the warning signal in the form of at least one of illuminating a warning light, outputting a voice, outputting a display signal, transmitting to an external control system, or notifying a driver in response to the risk level.

[0133] More specifically, as illustrated in FIG. 5, the object risk analysis unit (160) may include a behavior prediction unit (161) that predicts the entry intention of the target object based on the movement trajectory of the target object tracked through the object tracking unit (150), a collision prediction unit (162) that calculates the predicted movement path of the target object and determines whether the predicted movement paths overlap between the target objects, a collision time calculation unit (163) that calculates the expected collision time between the target objects based on at least one of the relative positional relationship of the movement paths between the target objects, the relative positional relationship of the predicted movement paths between the target objects, and the relative speed relationship between the target objects when there are multiple target objects whose predicted movement paths overlap in the collision prediction unit (162), and a warning control unit (164) that distinguishes a risk level according to the expected collision time and the type of the target object and outputs the warning signal in response to the risk level.

[0134] The above behavior prediction unit (161) can predict the future direction of the target object, the possibility of entering a specific area, etc., based on the movement trajectory, position change, direction of movement, speed of movement, and acceleration change of the target object tracked through the object tracking unit (150).

[0135] In addition, the behavior prediction unit (161) can analyze the temporal change of the object information tracked at consecutive points in time to determine whether the target object is likely to maintain its current moving state, change direction, or enter a pre-set danger zone, crossing zone, or vehicle path.

[0136] Accordingly, the behavior prediction unit (161) can not only determine the object's position at the current time but also analyze the potential future behavior of the target object in advance.

[0138] For example, the behavior prediction unit (161) can predict the entry intention of the target object by using at least one of the change amount between the current position and the previous position of the target object, the direction of movement vector, the change amount of movement speed, the change amount of acceleration, the change amount of rotation direction, and the change in walking posture.

[0139] For example, if a target object classified as a pedestrian approaches the boundary of the crossing area and a direction of travel is formed toward the roadway and the speed of movement increases, the behavior prediction unit (161) can determine that there is a high probability of entering the crossing.

[0140] In addition, if a target object classified as a vehicle or moving body approaches at high speed along a preset path, the behavior prediction unit (161) may designate it as a priority subject for collision risk analysis.

[0141] In addition, the behavior prediction unit (161) may apply different prediction criteria depending on the type of the target object.

[0142] For example, the behavior prediction unit (161) can predict the intention to enter more precisely by reflecting movement characteristic information corresponding to each type of object when there are different movement patterns for each type of object, such as pedestrians, bicycles, electric scooters, wheelchair users, or vehicles. Accordingly, the behavior prediction unit (161) can generate more reliable prediction results by reflecting speed ranges, turning characteristics, stopping-and-restart characteristics, or path deviation characteristics that appear differently depending on the type of object.

[0144] Meanwhile, the collision prediction unit (162) can calculate the predicted movement path of the target object and determine whether the predicted movement paths overlap between the target objects. More specifically, the collision prediction unit (162) can generate a predicted movement path that reflects the future movement direction and position change of each target object by comprehensively considering the entry intention predicted by the behavior prediction unit (161) and the movement trajectory tracked by the object tracking unit (150). At this time, the predicted movement path can be expressed in at least one of a path, an area, and a point sequence in which the target object is likely to move during a preset time interval.

[0145] For example, the collision prediction unit (162) can calculate the predicted movement path for each of the plurality of target objects, and then analyze whether the predicted movement paths intersect, overlap, or approach each other within a preset minimum separation distance to select pairs of objects that may collide.

[0146] For example, if the predicted movement path of the first target object and the predicted movement path of the second target object converge toward the same area or there is a possibility that they will pass through the same point within a certain time, the collision prediction unit (162) may determine this as a state where a collision possibility exists. Accordingly, the collision prediction unit (162) can reduce unnecessary operations on object pairs with a low probability of actual collision and set object pairs with a high probability of collision as priority analysis targets.

[0147] In addition, the collision prediction unit (162) does not limit the determination of whether the predicted movement paths overlap to a simple line segment intersection, but rather determines by reflecting the size of the target object, the occupied area, the deviation in the direction of movement, the possibility of speed change, or a preset safety margin section, thereby excluding formal path intersections unrelated to the actual collision risk and more accurately determining combinations of target objects that are substantially likely to collide.

[0149] Meanwhile, the collision time calculation unit (163) can calculate the expected collision time between the target objects based on at least one of the relative positional relationship of the movement paths between the target objects, the relative positional relationship of the predicted movement paths between the target objects, and the relative speed relationship between the target objects when there are multiple target objects whose predicted movement paths overlap in the collision prediction unit (162).

[0150] More specifically, the collision time calculation unit (163) can calculate the time at which different target objects are expected to reach the same point or the same area by comprehensively considering the location occupied by each target object at the current time, the location where future movement is expected, the direction of movement, and the relative speed.

[0151] For example, the collision time calculation unit (163) can calculate the expected collision time using the current separation distance and approach speed between a plurality of target objects, and may further reflect the remaining distance to the intersection point on the predicted movement path, path curvature, change in speed, and change in acceleration as needed.

[0152] In addition, the collision time calculation unit (163) can correct the expected collision time by reflecting such behavioral changes when there is a high probability that any of the target objects will decelerate, stop, or change direction.

[0153] In addition, the collision time calculation unit (163) may provide the calculated collision estimated time as a priority judgment factor for classifying the risk level if it is shorter than a preset reference time.

[0155] The warning control unit (164) can distinguish a risk level according to the expected collision time and the type of target object, and output a warning signal in response to the risk level.

[0156] More specifically, the warning control unit (164) can determine a higher risk level as the expected collision time is shorter, and even if the expected collision time is the same, if a target object with a high priority for protection, such as a pedestrian, child, elderly person, wheelchair user, or other vulnerable road user, is included, it can determine a higher risk level, thereby performing differential warning control that reflects the characteristics of the target object.

[0157] For example, the warning control unit (164) can divide the risk level into multiple stages, output a visual display or preliminary warning at a relatively low risk level, and output an active warning signal including a flashing light, voice guidance, alarm sound, transmission to an external control system, or driver notification at a high risk level.

[0158] Additionally, the warning control unit (164) may stop the warning signal or switch to a lower level warning state when the danger level is resolved or the possibility of collision between target objects is eliminated, and accordingly, the warning control unit (164) can provide a quick and clear warning in actual danger situations while reducing unnecessary false alarms.

[0160] That is, the object risk analysis unit (160) predicts the entry intention of a target object through the behavior prediction unit (161), determines whether the predicted movement paths between multiple target objects overlap through the collision prediction unit (162), calculates the expected collision time between target objects through the collision time calculation unit (163), and outputs a response warning signal by distinguishing the risk level according to the expected collision time and the type of target object through the warning control unit (164), thereby supporting a preemptive and precise response to an actual dangerous situation.

[0162] Accordingly, the radar-camera sensor fusion-based intelligent object tracking system (100) according to one embodiment of the present invention aligns camera-based image information and radar-based physical information spatially and temporally to fuse them together, improves computational efficiency through radar-based area of ​​interest setting, and performs classification of target objects, tracking of movement status, prediction of entry intention, and collision risk analysis in conjunction, thereby enabling more stable tracking of target objects even in various lighting and weather environments and providing warning signals at appropriate times.

[0164] Meanwhile, as another example, the object risk analysis unit (160) calculates the collision risk integrated index R based on the following [Mathematical Formula 1] in order to quantitatively calculate the collision risk between target objects. cf It can produce.

[0166] [Mathematical Formula 1]

[0167]

[0168] Here, R cf is a composite collision risk index representing the collision risk between target objects, and W ot is an object type weight assigned according to the type of the target object, and C so is the same object judgment reliability calculated by the same object judgment unit (151), and S stis the spacetime alignment reliability calculated by the spacetime synchronization unit (130), and P ov is the predicted movement path overlap calculated by the collision prediction unit (162), and V rel is the relative velocity value between multiple target objects, and D rel is a relative distance value between multiple target objects, and T col is the expected collision time calculated by the collision time calculation unit (163), and α, β and γ represent weighting factors for adjusting the reflection ratio of each risk factor.

[0170] More specifically, W in the above [Equation 1] ot , C so and S st Each can function as a correction term reflecting the type of target object, the reliability of identical object determination, and the reliability of spatiotemporal alignment.

[0171] Also, P inside the parentheses ov , V rel / D rel and 1 / T col Each can function as a risk calculation term reflecting the potential collision probability due to path overlap, the degree of approach relative distance relative to relative speed, and the urgency of the time remaining until collision.

[0172] Accordingly, the object risk analysis unit (160) can calculate the collision risk by comprehensively considering a plurality of technical factors derived from the sensor fusion structure of the present invention, rather than a simple distance-based judgment.

[0174] At this time, the above relative distance D rel and the above collision estimated time T col can be defined as a non-zero value, and the relative distance D rel or the above collision estimated time T colIf this is calculated to be 0 or converges to 0, it can be calculated by correcting it to a preset minimum reference value. Therefore, the above [Equation 1] can be calculated stably without the denominator becoming 0.

[0176] Meanwhile, the above-mentioned spatiotemporal alignment reliability S st represents a value reflecting the accuracy of spatial alignment and temporal alignment performed by the spatiotemporal synchronization unit (130). The spatiotemporal alignment reliability S st The reason for reflecting this separately is that if the radar data and the image data are not sufficiently aligned on the same reference coordinate system and the same reference point, the reliability of the same object judgment or collision prediction result may also be reduced.

[0177] For example, the above-mentioned spatiotemporal alignment reliability S st It can be calculated based on the following [Mathematical Formula 2].

[0179] [Mathematical Formula 2]

[0180]

[0181] Here, E sp is a spatial alignment error value based on the correspondence relationship between 2D image coordinates and 3D spatial coordinates, and E tm is a time alignment error value based on the difference between the exposure time and the scan time, and λ1 and λ2 represent weighting factors to adjust the effect of spatial alignment error and time alignment error on spatiotemporal alignment reliability.

[0183] More specifically, the above spatial alignment error value E sp can be calculated based on the difference between the coordinates obtained by projecting the radar coordinates onto the image plane and the center coordinates or reference coordinates of the actual object candidate region. In addition, the time alignment error value E mentioned above tmIt can be calculated based on the time difference between the camera's exposure time and the radar's scan time, or the time difference between the virtual radar data generated by interpolation and the actual camera frame time.

[0184] According to the above [Mathematical Formula 2], the spatial alignment error value E sp and time alignment error value E tm The smaller α is, the greater the spacetime alignment reliability S st can be calculated as a large value, and conversely, as the above error values ​​increase, the spatiotemporal alignment reliability S st can be reduced. Accordingly, the above-mentioned spatiotemporal alignment reliability S st is the above-mentioned collision risk integrated index R cf It can be reflected as a consistency correction factor.

[0186] In addition, the above predicted movement path overlap degree P ov is a value calculated by the collision prediction unit (162) and represents a value that quantitatively indicates how much the predicted movement paths between multiple target objects spatially overlap. The reason for separately reflecting the degree of overlap of the predicted movement paths is that even if the relative speed between target objects is high, the actual risk of collision may be low if the actual movement paths do not intersect or overlap.

[0187] For example, the above predicted movement path overlap degree P ov It can be calculated based on the following [Mathematical Formula 3].

[0189] [Mathematical Formula 3]

[0190]

[0191] Here, A ov is the size of the overlapping area where the predicted movement path area of ​​the first target object and the predicted movement path area of ​​the second target object overlap, and A obj1 is the size of the predicted movement path area of ​​the first target object, and A obj2 represents the size of the predicted movement path area of ​​the second target object.

[0193] More specifically, the collision prediction unit (162) can generate a predicted movement path of each target object in the form of an area, and then calculate the size of the portion where the predicted movement path areas between multiple target objects overlap. At this time, the size A of the overlapping area ov The larger α is and the higher the overlap ratio relative to the predicted movement path area of ​​each object, the greater the above predicted movement path overlap P ov can be calculated as a larger value. Conversely, if the predicted movement paths are separated from each other or the overlap area is very small, the above predicted movement path overlap degree P ov can be calculated as a small value. Accordingly, the above predicted movement path overlap P ov It can be used as a key factor to select pairs of objects with a high probability of actual collision.

[0194] At this time, the size A of the predicted movement path area of ​​the first target object. obj1 , size A of the predicted movement path area of ​​the second target object above obj2 and the size A of the above overlapping area ov can be defined as a value greater than or equal to 0, and the predicted movement path area of ​​the first target object and the predicted movement path area of ​​the second target object can be set to include at least some valid area.

[0196] That is, the object risk analysis unit (160) according to one embodiment of the present invention uses [Equation 1] to [Equation 3] to organically reflect the accuracy of spatiotemporal alignment, the reliability of identical object judgment, the degree of overlap of predicted movement paths, relative distance, relative speed, and the expected collision time, and the collision risk integrated index R cf It can produce.

[0197] Accordingly, the present invention can perform a more precise and reliable risk analysis by comprehensively reflecting multiple technical factors derived from the entire sensor fusion process, rather than relying on simple object detection results.

[0199] Meanwhile, the object risk analysis unit (160) calculates the collision risk integrated index R according to [Equation 1]. cf The risk level can be distinguished using [this]. For example, the warning control unit (164) of the object risk analysis unit (160) uses the collision risk integrated index R cf If the value is less than the first threshold value, it can be classified as a low-risk stage; if the value is greater than or equal to the first threshold value but less than the second threshold value, it can be classified as a medium-risk stage; and if the value is greater than or equal to the second threshold value, it can be classified as a high-risk stage. In this case, the first threshold value and the second threshold value may be variably set according to the installation environment, characteristics of the monitored area, type of target object, or required safety level.

[0201] More specifically, at the low-risk stage, a warning at the level of a visual display or preliminary guidance can be output, at the medium-risk stage, voice guidance or warning sound output can be performed, and at the high-risk stage, active warnings including flashing lights, high-output alarms, transmission to an external control system, or driver notification can be output.

[0202] As such, the above-mentioned collision risk integrated index R cf If used as a multi-stage risk assessment criterion, it is possible to control the warning intensity in proportion to the actual risk level, and it is possible to reduce unnecessary false alarms while enabling a rapid response to actual risk situations.

[0204] <Example 2>

[0205] Meanwhile, the radar-camera sensor fusion-based intelligent object tracking method of the radar-camera sensor fusion-based intelligent object tracking system (100) according to an embodiment of the present invention comprises: a first step in which the image data collection unit (110) collects real-time image data from at least one camera terminal; a second step in which the radar data collection unit (120) collects radar data including at least one of distance information, azimuth information, and speed information from at least one radar sensor; a third step in which the spatiotemporal synchronization unit (130) corresponds the real-time image data and the radar data to a preset identical reference point and a preset identical reference coordinate system to generate synchronization data; a fourth step in which the interest region setting unit (140) extracts at least one object candidate region from the real-time image data based on the synchronization data and sets an interest region corresponding to the object candidate region; a fifth step in which the object tracking unit (150) classifies a target object within the interest region and tracks the movement state of the target object; and an object risk analysis unit (160) analyzes the movement trajectory of the target object to calculate the possibility of collision risk and the It may include a sixth step of outputting a warning signal when the risk of collision is greater than or equal to a preset threshold.

[0206] For example, in the first step, the image data collection unit (110) may collect real-time image data of a surveillance target area from at least one camera terminal installed as a fixed or mobile type. At this time, the real-time image data may be provided as a continuous image of multiple frames, and each frame may be accompanied by shooting time information, frame number, exposure time information, or camera internal parameter information. Additionally, in the first step, at least one of distortion correction, noise removal, contrast correction, or resolution adjustment may be performed to increase the reliability of subsequent object candidate area extraction.

[0208] Additionally, in the second step, the radar data collection unit (120) may collect radar data including at least one of distance information, azimuth information, and velocity information from at least one radar sensor. The radar data may further include reflection intensity information, Doppler information, cluster center information, or point cloud information as needed. Furthermore, in the second step, scan time information or timestamp information corresponding to each radar data may be generated together and utilized in a subsequent spatiotemporal synchronization step.

[0210] Meanwhile, in the third step, the spatiotemporal synchronization unit (130) may calculate a transformation matrix based on the correspondence between the 2D image coordinates extracted from the real-time image data and the 3D spatial coordinates extracted from the radar data, and perform spatial alignment using the transformation matrix. Additionally, in the third step, temporal alignment may be performed to correct the difference between the exposure time extracted from the real-time image data and the scan time extracted from the radar data. Accordingly, in the third step, the synchronization data may be generated so that the real-time image data and the radar data correspond on the same reference time and the same reference coordinate system.

[0212] For example, in the fourth step, the interest region setting unit (140) may extract at least one object candidate region from the real-time image data based on the synchronization data and generate image feature data corresponding to the object candidate region. Additionally, the size of the interest region may be determined based on the size of the object candidate region, and the center coordinates of the interest region may be calculated by projecting the radar data onto the planar coordinates of the real-time image data. Furthermore, if the distance between the object candidate regions is less than a preset reference distance, the object candidate regions may be grouped into a single object unit interest region to improve the efficiency of subsequent operations.

[0214] In addition, in the fifth step, the object tracking unit (150) can determine whether the objects are the same and generate object information based on image feature data corresponding to the object candidate area included within the area of ​​interest and the radar data. In the object tracking step, the type of the target object can be classified based on the object information, and the location change and direction of movement of the target object can be tracked based on the temporal change of the object information. Accordingly, in the fifth step, even when multiple target objects exist simultaneously, the movement trajectory of each target object can be continuously managed.

[0216] Meanwhile, in the sixth step, the object risk analysis unit (160) can predict the entry intention of the target object based on the movement trajectory of the target object tracked in the fifth step. Additionally, it can calculate the predicted movement path of the target object and determine whether the predicted movement paths overlap between the target objects. If there are multiple target objects whose predicted movement paths overlap, the expected collision time between the target objects can be calculated based on at least one of the relative positional relationship of the movement paths between the target objects, the relative positional relationship of the predicted movement paths between the target objects, and the relative speed relationship between the target objects.

[0217] In addition, in the sixth step, a danger level can be classified according to the estimated collision time and the type of the target object, and a warning signal including at least one of illuminating a warning light, voice guidance, outputting an alarm sound, transmitting to an external control system, or notifying a driver can be output in response to the danger level.

[0218] Accordingly, the above-described radar-camera sensor fusion-based intelligent object tracking method can go beyond simple object detection, track the movement status of multiple target objects, and proactively analyze and respond to actual collision risk situations.

[0219] That is, the radar-camera sensor fusion-based intelligent object tracking method according to one embodiment of the present invention aligns camera-based image information and radar-based physical information spatiotemporally to fuse them together, improves computational efficiency through object candidate region extraction and region of interest setting, and performs classification of target objects, tracking of movement status, prediction of entry intention, and collision risk analysis in conjunction, thereby enabling stable tracking of target objects even in various lighting and weather environments and providing warning signals at appropriate times.

[0221] According to the present invention as described above, a radar-camera sensor fusion-based intelligent object tracking system and method can be provided, which can fuse data obtained from different sensors in a consistent manner by synchronizing real-time image data collected from a camera terminal and radar data collected from a radar sensor spatiotemporally on the same reference point and the same reference coordinate system.

[0222] In addition, an intelligent object tracking system and method based on radar-camera sensor fusion can be provided, which can improve the efficiency of object recognition while reducing unnecessary computations by extracting object candidate regions from real-time image data based on spatiotemporally aligned synchronization data and setting corresponding regions of interest.

[0223] In addition, an intelligent object tracking system and method based on radar-camera sensor fusion can be provided, which can more accurately recognize target objects in various environments by combining image feature data and radar data within a region of interest to determine whether they are the same object and classifying the type of target object.

[0224] In addition, a radar-camera sensor fusion-based intelligent object tracking system and method can be provided, which can stably track the continuous movement state of multiple target objects by tracking the position change and movement direction of the target objects based on the object information and temporal change of the movement trajectory of the classified target objects.

[0225] In addition, an intelligent object tracking system and method based on radar-camera sensor fusion can be provided, which can predict the entry intention of a target object and calculate the expected collision time based on whether the predicted movement paths between target objects overlap, relative positional relationships, and relative speed relationships, thereby analyzing the collision risk and outputting a warning signal corresponding to the risk level to induce the prevention of dangerous accidents.

[0227] In addition, a control method for an intelligent object tracking system based on radar-camera sensor fusion according to an embodiment of the present invention may be recorded on a computer-readable medium containing program instructions for performing operations implemented by various computers. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions on the medium may be those specifically designed and configured for the present invention, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0229] As described above, although an embodiment of the present invention has been explained by limited embodiments and drawings, the embodiment of the present invention is not limited to the embodiments described above, and various modifications and variations are possible from this description by those skilled in the art to which the present invention pertains. Accordingly, an embodiment of the present invention should be understood only by the claims described below, and all equivalent or analogous variations thereof shall be considered to fall within the scope of the inventive concept. Explanation of the symbols

[0231] 110: Image data acquisition unit 120: Radar Data Acquisition Unit 130 : Spacetime Synchronization Unit 131 : Space alignment section 132 : Time alignment section 140 : Region of Interest Setting Section 141 : Object Candidate Extraction Unit 142 : Variable setting section for region of interest 143 : Projection Mapping Section 144 : Computation Optimizer 150 : Object tracking section 151 : Same object determination unit 152 : Object Classification Unit 153 : Movement status tracking section 160 : Object Risk Analysis Department 161 : Behavior Prediction Section 162 : Collision Prediction Section 163: Collision Time Calculation Unit 164 : Warning Control Unit

Claims

Claim 1 A video data collection unit for collecting real-time video data from at least one camera terminal; a radar data collection unit for collecting radar data including at least one of distance information, azimuth information, and speed information from at least one radar sensor; a spatiotemporal synchronization unit for generating synchronization data by aligning the real-time video data collected through the video data collection unit and the radar data collected through the radar data collection unit with a preset identical reference point and a preset identical reference coordinate system; an interest area setting unit for extracting at least one object candidate region from the real-time video data based on the synchronization data aligned by the spatiotemporal synchronization unit and setting an interest area corresponding to the object candidate region; and an object tracking unit for classifying a target object within the interest area set by the interest area setting unit and tracking the movement state of the target object. and an object risk analysis unit that analyzes the movement trajectory of the target object classified by the object tracking unit to calculate the probability of collision risk of the target object, and outputs a warning signal if the probability of collision risk is greater than or equal to a preset threshold; wherein the spatiotemporal synchronization unit includes a spatial alignment unit that calculates a transformation matrix and performs spatial alignment based on the correspondence relationship between 2D image coordinates extracted from the real-time video data and 3D spatial coordinates extracted from the radar data, and calculates the transformation matrix by matching the 2D image coordinates of a reference point detected from the real-time video data with the 3D spatial coordinates of a corresponding reference point obtained from the radar data;The method includes a time alignment unit that corrects the difference between the exposure time extracted from the real-time image data and the scan time extracted from the radar data, sets a reference time by comparing the timestamps assigned to the real-time image data and the radar data, and calculates radar position information of the target object corresponding to the frame acquisition time of the camera terminal by applying at least one correction among linear interpolation, curve interpolation, and prediction model-based correction; wherein the interest region setting unit comprises: an object candidate extraction unit that extracts at least one object candidate region from the real-time image data and generates image feature data corresponding to the object candidate region; an interest region variable setting unit that determines the size of the interest region based on the size of the object candidate region; and a projection mapping unit that calculates the center coordinates of the interest region by projecting the radar data onto the planar coordinates of the real-time image data. and if the distance between the object candidate regions is less than a preset reference distance, the operation optimization unit groups the object candidate regions into a single object unit interest region; the object tracking unit includes: an identical object determination unit that determines whether the objects are identical and generates object information based on image feature data corresponding to the object candidate regions included within the interest region and the radar data; an object classification unit that classifies the type of the target object based on the object information generated through the identical object determination unit; and a movement state tracking unit that tracks the position change and movement direction of the target object based on the temporal change of the object information; and the object risk analysis unit includes: a behavior prediction unit that predicts the entry intention of the target object based on the movement trajectory of the target object tracked through the object tracking unit; and a collision prediction unit that calculates the predicted movement path of the target object and determines whether the predicted movement paths overlap between the target objects.A collision time calculation unit that calculates an expected collision time between target objects based on at least one of the relative positional relationship of the movement paths between target objects, the relative positional relationship of the predicted movement paths between target objects, and the relative speed relationship between target objects when there are multiple target objects whose predicted movement paths overlap in the collision prediction unit; and a warning control unit that classifies a risk level according to the expected collision time and the type of target object, and outputs a warning signal corresponding to the risk level; wherein the object risk analysis unit includes an integrated collision risk index R based on the following [Equation 1]; cf Calculate and, [Mathematical Formula 1] (here, R cf is a composite collision risk index representing the collision risk between target objects, and W ot is an object type weight assigned according to the type of the target object, and C so is the same object judgment reliability calculated by the same object judgment unit (151), and S st is the spacetime alignment reliability calculated by the spacetime synchronization unit (130), and P ov is the predicted movement path overlap degree calculated by the above collision prediction unit, and V rel is the relative velocity value between multiple target objects, and D rel is a relative distance value between multiple target objects, and T col is the estimated collision time calculated by the collision time calculation unit, and α, β, and γ represent weighting factors for adjusting the reflection ratio of each risk factor) The warning control unit, the collision risk integrated index R cf The above risk levels are classified based on, wherein the above collision risk integrated index R cf A radar-camera sensor fusion-based intelligent object tracking system characterized by: determining a low-risk stage when the value is less than a first threshold value and outputting the warning signal including the illumination of a warning light corresponding to a visual display; determining a medium-risk stage when the value is greater than or equal to a second threshold value and outputting the warning signal including at least one of voice guidance and warning sound output; determining a high-risk stage when the value is greater than or equal to the second threshold value and outputting the warning signal including the illumination of the warning light and the high-power alarm; and transmitting a notification signal to a preset external control system and a preset driver terminal. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 delete

Citation Information

Patent Citations

  • Calibration apparatus and method of camera and rader

    KR102016636B1

  • Mechanisms to improve image capture operations

    KR1020230091097A

  • Vehicle, method of control vehicle, and driving assistance system of vehicle

    KR1020230141982A

  • System and method for providing abnormal event information in a road traffic area by integratedly analyzing multi-sensor collected information based on artificial intelligence

    KR102978377B1

  • Adaptive bounding box merge method in blob analysis for video analytics

    US20180047193A1