Apparatus and Method for Predicting Vehicle-Pedestrian Collisions Using Weighted Integrated Feature Map Generation and Risk Index Calculation

Abstract

The present invention relates to a multimodal sensor fusion-based pedestrian detection and prediction system (100) and a method for detecting pedestrians or two-wheeled vehicles around a vehicle and predicting collision risk, wherein a plurality of multimodal sensors having different detection principles are mounted on a vehicle. A specific feature of the present invention is that it includes: a sensor module (110) positioned outside the vehicle to collect environmental data around the vehicle; a multimodal fusion unit that aligns data from different modal sensors collected from the sensor module (110) with a common reference time and vehicle reference coordinate system, and generates an integrated feature map by applying reliability-based weights of each sensor; an AI-based recognition and prediction unit (130) that receives the integrated feature map and calculates object recognition, object tracking, pedestrian crossing intention prediction, and collision risk; and a vehicle integrated terminal (140) that determines a final risk level by combining analysis values ​​including risk, TTC, and object trajectory calculated by the AI ​​recognition and prediction unit (130) with vehicle ECU information.

Description

Technology Field

[0001] The present invention relates to a technology for detecting pedestrians and two-wheeled vehicles (including PMs) around a vehicle and predicting the risk of collision in real time. More specifically, the invention relates to a system and method for performing 360° all-around pedestrian recognition, adverse condition response recognition, and on-device AI-based collision risk prediction by fusing heterogeneous sensors such as RGB, IR, ultrasonic, and LiDAR. Background Technology

[0002] Collisions with pedestrians or motorcycles while driving are one of the most frequent types of accidents in urban traffic environments, and the risk of accidents caused by blind spots is continuously reported, particularly in heavy vehicles such as large trucks, commercial cargo trucks, and buses. To address this problem, Advanced Driver Assistance Systems (ADAS) are widely applied to vehicles, and certain functions are provided, including front camera and radar-based Forward Collision Warning (FCW), Automatic Emergency Braking (AEB), Blind Spot Detection (BSD), and Lane Departure Warning (LDW).

[0003] For example, Patent Application No. 10-2017-0009173 (Title of Invention: “Camera System and Emergency Braking System and Method for ADAS”) also provides a forward recognition function based on camera images and discloses a technology that performs emergency braking by detecting obstacles or pedestrians in front. However, most of these prior art technologies are based on a forward-centered field of view, providing only limited recognition of the vehicle's driving direction and having limitations in adequately monitoring wide blind spots such as the left, right, and rear sides of the vehicle. In particular, commercial vehicles such as large buses, trucks, and delivery vehicles often have narrow side or rear fields of view due to their body structure, and the high position of the driver's seat frequently prevents pedestrians or personal mobility devices (PMs) from entering the driver's direct field of view. Consequently, it is difficult to fully capture actual hazards around the vehicle using only a front camera or a single radar, resulting in a problem where accident prevention performance is limited.

[0004] In addition, conventional image-based ADAS has a fundamental limitation that makes it vulnerable to environmental changes.

[0005] For example, at night, the signal-to-noise ratio (SNR) of RGB cameras drops sharply due to insufficient illumination, and in backlit environments, the outline of an object disappears, making detection difficult. In weather conditions such as rain, snow, and fog, camera image contrast is lowered and the lidar signal is attenuated, and image quality is degraded in shades of tunnels, underpasses, and overpasses.

[0006] In such situations, existing ADAS algorithms frequently cause false positives by misidentifying people as objects or false negatives by failing to detect people at all, significantly compromising safety.

[0007] Conventional technology also has limitations in terms of sensor configuration. Single-camera based systems have low distance estimation accuracy, which restricts the calculation of actual collision risks with pedestrians, while single-radar based systems struggle to distinguish multiple objects in confined spaces due to poor angular resolution. Although LiDAR-based systems enable relatively accurate distance measurement, their high sensor prices and maintenance costs make it difficult to expand their application in cost-sensitive industries such as commercial vehicles and light cargo trucks. In particular, the reality is that the adoption of LiDAR is limited by cost issues in most general vehicle markets, excluding autonomous vehicles.

[0008] Meanwhile, due to changes in the traffic environment and the increase in urban density, pedestrian behavior around vehicles is becoming more complex. Because of various variables, such as the "Smombie" phenomenon (pedestrians driving while looking at their smartphones), the rise in PM users, and sudden pedestrian entry, traditional methods that merely detect pedestrian locations are insufficient to adequately prevent collisions. Existing ADAS systems lack the capability to deeply analyze pedestrians' crossing intentions or behavioral patterns; consequently, they fail to reflect the fundamental elements of dangerous situations, leading to a problem where the accuracy of warnings is low.

[0009] In addition, since Time-To-Collision (TTC)-based risk calculations only consider relative speed and distance, they fail to reflect whether a pedestrian actually intends to cross the road (intention). Consequently, unnecessary warnings may be issued even in situations where there is no actual danger, or conversely, situations where a pedestrian is running towards the road may not be detected in time. As a result, drivers may lose trust in the system due to excessive warnings, or there is a risk of accidents occurring because warnings are not provided at moments when they are actually needed.

[0010] As such, conventional technology has various limitations and consequently fails to adequately reflect dangerous situations occurring in actual road environments. Therefore, there is a need for environmentally adaptive sensor fusion technology capable of fully detecting the 360° surroundings of a vehicle and automatically adapting to various environmental changes. Furthermore, there is a need for technology that satisfies accuracy, stability, and low cost by simultaneously utilizing multiple sensors with different detection principles rather than a single sensor. Moreover, there is an urgent need for an AI-based collision risk prediction system that can predict actual dangerous situations in real time with high reliability by comprehensively considering pedestrian behavioral intentions and TTC information. The problem to be solved

[0011] As mentioned above, conventional ADAS technology has structural limitations, including a limited forward-focused field of view, degraded recognition performance in adverse conditions, false alarms and failure to detect due to a single-sensor structure, high costs associated with LiDAR and radar-based systems, and the inability to comprehensively consider pedestrian behavioral intentions and TTC. Therefore, the present invention aims to solve the following technical challenges to overcome these problems of conventional technology and achieve the safety level required in actual road environments.

[0012] The problem to be solved by the present invention is to provide a technology capable of reliably detecting a 360° all-around area around a vehicle. While conventional front camera or radar-centered ADAS focuses only on the front of the vehicle in the direction of travel, the present invention enables all-around monitoring including side and rear blind spots through multiple sensors positioned on the front, rear, left, and right sides of the vehicle, thereby aiming to fundamentally reduce blind spot accidents that frequently occur, particularly in large vehicles or commercial vehicles.

[0013] Furthermore, another objective of the present invention is to provide an environment-adaptive multimodal sensor fusion technology capable of maintaining stable recognition performance even under various environmental changes, such as day and night, low light, backlight, rain, and fog. To this end, the invention goes beyond simply merging data collected from sensors with different detection principles, such as RGB cameras, IR cameras, ultrasonic sensors, and LiDAR; instead, it fuses the data at the pixel level based on reliability weights that reflect the quality, visibility, reliability, and time-series stability of each sensor, thereby ensuring the recognition stability of the entire system even under adverse conditions where the performance of a specific sensor deteriorates.

[0014] In addition, another objective of the present invention is to precisely perform temporal and spatial alignment between sensors so that heterogeneous sensor data having different frame rates and installation locations can be interpreted in a single common viewpoint and coordinate system. That is, the purpose is to provide an implementation structure that can reproduce accurate multimodal fusion results by applying the same algorithm, by reflecting egomotion information such as vehicle speed, steering angle, and acceleration to correct the observation point of sensor data to a reference time, and by integrating the coordinate systems of each sensor into a vehicle reference coordinate system and a panoramic image coordinate system.

[0015] Furthermore, another objective of the present invention is to provide an AI-based prediction technology capable of predicting actual collision risks with high reliability by simultaneously considering not only the pedestrian's location and speed information but also the pedestrian's behavioral intent and Time-to-Collision (TTC). To this end, the invention aims to implement a system that moves beyond simple distance- and speed-based alerts and responds more closely to actual dangerous situations by calculating a Risk Index (RR) through the combination of an intent prediction model, which learns time-series information such as the pedestrian's head direction, body rotation, movement speed, relative distance, and distribution of surrounding pedestrians, and TTC calculations, and by generating an alert based on this Risk Index.

[0016] In addition, another objective of the present invention is to provide a low-latency, low-power AI computing structure that enables such advanced recognition and prediction functions to be performed in real-time in an on-device (Edge) environment within a vehicle without relying on a cloud server. By utilizing FPGA or ASIC-based parallel processing and pipeline structures to minimize the overall recognition and prediction latency while reducing power consumption, the invention aims to realize a structure that can be economically applied to various vehicle platforms, such as commercial vehicles and light cargo trucks.

[0017] Furthermore, another objective of the present invention is to provide a pedestrian detection and prediction solution with excellent cost-effectiveness by utilizing a combination of RGB cameras, IR cameras, ultrasonic sensors, and optional LiDAR, along with reliability-based fusion, rather than relying solely on expensive LiDAR and radar systems. Through this, the aim is to secure economic feasibility that allows for practical implementation in cost-sensitive vehicle groups, such as general passenger cars, light trucks, delivery vehicles, and buses, rather than being limited to expensive vehicles like autonomous cars.

[0018] Furthermore, another objective of the present invention is to enable bidirectional safety by combining the prediction results with vehicle ECU and road infrastructure information to provide intuitive visual, auditory, and tactile warnings to both drivers and pedestrians. In other words, the ultimate goal is to implement a new type of pedestrian safety system that allows both vehicles and pedestrians to actively respond to danger by clearly communicating dangerous situations to drivers through the HUD, cluster, warning sounds, and vibrations, and by notifying pedestrians of approaching vehicles and dangerous situations through external LED panels, flashing lights, and voice guidance. means of solving the problem

[0019] The solution means of the present invention for solving the above problem is a multimodal sensor fusion-based pedestrian detection and prediction system that predicts the risk of collision of pedestrians or two-wheeled vehicles around a vehicle using a plurality of sensors having different detection principles, comprising: a sensor module unit disposed outside the vehicle and including an RGB camera, an IR camera, and an ultrasonic sensor, and collecting environmental data around the vehicle; and for the sensor data collected from the sensor module unit, (b1) time alignment to a common reference time, (b2) spatial alignment to a vehicle reference coordinate system, (b3) illuminance, presence of rainfall, frame quality (q s ), Visibility(v s ), measurement reliability (c s ), time series consistency(m s ), outlier suppression(o sA multimodal fusion unit that generates an integrated feature map by calculating at least one environment-adaptive weight among ) and weighting the feature values ​​per sensor at the pixel or spatial grid level; an AI-based recognition and prediction unit that calculates a crossing intention probability (P_cross) based on the integrated feature map using behavior-based features including at least one of (c1) object recognition, (c2) time-series-based object tracking, (c3) pedestrian head direction, body rotation angle, speed change, and surrounding pedestrian distribution, and (c4) calculates a risk index (RR) defined as RR = P_cross / TTC by combining the P_cross and TTC (Time-To-Collision); and a vehicle integrated terminal and a warning display unit that determine a risk level based on the risk index (RR) and output a warning signal to a driver or pedestrian.

[0020] In addition, in the present invention, it is preferable that the multimodal fusion unit further includes an environment condition-based real-time weight boost rule that increases the weight of the IR camera in night or low-light environments, increases the weight of the ultrasonic sensor when rain or snow is detected, and decreases the weight of the RGB camera when backlight or glare is detected.

[0021] In addition, in the present invention, the AI-based recognition and prediction unit preferably includes a deep learning-based time series prediction model that takes as input at least one time series feature among a change in head direction (Δθ), body rotation angle, change in speed or acceleration, relative angle with respect to the vehicle approach direction, and the location and density of surrounding pedestrians in order to calculate the probability of a pedestrian's crossing intention (P_cross).

[0022] In addition, in the present invention, the AI-based recognition and prediction unit is configured with a parallel streaming pipeline structure based on an FPGA or ASIC, and preferably includes an on-device AI computing unit configured to process CNN-based object recognition, BiLSTM or GRU-based time series analysis, Time-To-Collision (TTC) calculation, and Risk Index (RR) calculation within a preset delay time.

[0023] In addition, in the present invention, it is preferable that the vehicle integrated terminal is configured to collect vehicle speed, steering angle, braking pressure, lane information, and road infrastructure (SPaT, I2V) information, and to perform at least one of outputting a driver's HUD or cluster warning, outputting a warning sound or vibration, or determining whether to execute automatic vehicle braking control (AEB) by combining this with a risk index (RR) calculated by the AI-based recognition and prediction unit.

[0024] In addition, another solution of the present invention relates to a method for predicting the risk of collision of pedestrians or two-wheeled vehicles around a vehicle, comprising: (a) collecting environmental data from multimodal sensors including an RGB camera, an IR camera, and an ultrasonic sensor; (b) for the collected data, (b1) aligning with a common reference time, (b2) converting to a vehicle reference coordinate system, and (b3) illuminance, presence of rainfall, frame quality (q s ), Visibility(v s ), measurement reliability (c s ), time series consistency(m s ), outlier suppression(o s(a) a step of generating an integrated feature map by weighting the feature values ​​per sensor in pixel or spatial grid units based on at least one weighting factor among (a) and generating an integrated feature map; (c) a step of recognizing an object and performing time-series tracking based on the integrated feature map, and calculating a probability of crossing intent (P_cross) using at least one time-series information among head direction, body rotation, speed, and surrounding pedestrian distribution; (d) a step of calculating a risk index (RR) using the formula RR = P_cross / TTC using the P_cross and TTC (Time-To-Collision); (e) a step of providing an alert to a driver or pedestrian according to the calculated risk index (RR).

[0025] In addition, in the present invention, it is preferable to generate the integrated feature map by reflecting an environment-adaptive weight boost rule, wherein the weight calculated in step (b3) includes increasing the IR camera weight in nighttime or low light conditions, increasing the ultrasonic sensor weight in rain or snow conditions, and decreasing the RGB camera weight in backlight conditions. Effects of the invention

[0026] The multimodal sensor fusion-based pedestrian detection and prediction system according to the present invention integrates data acquired from multiple sensors having different detection principles, such as RGB cameras, IR cameras, ultrasonic sensors, and LiDAR, in time and space without relying on a single sensor for the vehicle's surrounding environment, thereby enabling stable and accurate 360° surrounding recognition in all environments. Accordingly, it is possible to fundamentally resolve the limitations of existing ADAS, such as limited field of view, failure to recognize in adverse conditions, and false alarms based on single sensors.

[0027] The present invention maintains stable recognition performance even under adverse conditions such as daytime, nighttime, backlight, rain, and fog by aligning and normalizing data collected from multiple sensors into a common reference time and a vehicle reference coordinate system, and then fusing them at the pixel level according to reliability weights for each sensor.

[0028] Quality of each sensor (q s ), Visibility(v s ), measurement reliability (c s ), time series consistency(m s By dynamically evaluating and adjusting weights in real time, robustness can be ensured so that the performance of the entire system does not deteriorate even if the performance of one sensor temporarily degrades. This effectively prevents the problems of non-detection and misrecognition that occur in conventional single-camera-based systems.

[0029] In addition, the present invention can quantitatively predict a pedestrian's future movement path and crossing intention by performing object recognition, object tracking, pedestrian intention prediction, and collision risk analysis stepwise using a deep learning model (CNN, LSTM, GRU, etc.) based on an integrated feature map.

[0030] In particular, since it calculates P_cross (crossing probability) by learning behavior-based time series features such as the pedestrian's head direction, body rotation, movement speed, and relative distance, it enables situation prediction with much higher reliability than simple location-based alerts. This effectively resolves the problem where existing ADAS frequently issued unnecessary alerts because it failed to consider the pedestrian's behavioral intentions or future paths.

[0031] In addition, the present invention can provide a warning based on the driver's actual accident risk by calculating a risk index (RR) by combining the Time-To-Collision (TTC) and the probability of crossing intent (P_cross). Therefore, unnecessary warnings are reduced in situations where the TTC is short but the probability of intent is low, or where the probability of intent is high but the TTC is sufficiently long, and a rapid and strong warning is output in truly dangerous situations. This is a major effect that simultaneously solves the biggest problems of existing systems, namely "over-alerting" and "false negatives."

[0032] In addition, the present invention adopts an FPGA / ASIC-based on-device AI structure, enabling all computations to be performed inside the vehicle with a delay of 50ms or less without connecting to a cloud or external server. This allows the system to operate normally even in communication dead zones or tunnels, and offers significant advantages in terms of personal information protection as personal video data is not transmitted externally.

[0033] In addition, the present invention can be linked with a vehicle ECU, braking system, and steering system through a vehicle integrated terminal (140), and can additionally collect external information such as intersection signals (SPaT) and pedestrian detection infrastructure using I2V communication. This enables the detection of the presence of pedestrians even in non-line-of-sight (NLOS) situations, thereby providing a blind spot pedestrian detection function that was impossible with existing vehicle-only sensor-based technologies.

[0034] In addition, the present invention provides a two-way safety function that simultaneously protects the driver (HUD, cluster, sound, touch) and the pedestrian (front LED panel, warning light, voice warning) through the warning display unit (150). This has significant technical and industrial effects in that it overcomes the limitations of existing ADAS, which were functions only for the driver, and implements a new concept of vehicle-pedestrian safety HMI that directly notifies pedestrians of dangerous situations. Brief explanation of the drawing

[0035] Figure 1 is an overall configuration diagram illustrating the overall system of the present invention. Figure 2 is a block diagram of the prediction sequence of the AI-based recognition and prediction unit of the present invention. FIG. 3 is a flowchart illustrating the overall execution steps of the present invention. Specific details for implementing the invention

[0036] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0037] FIG. 1 is an overall configuration diagram illustrating the overall system of the present invention, and FIG. 2 is a block diagram of the prediction sequence of the AI-based recognition and prediction unit of the present invention.

[0038] The 'multimodal sensor fusion-based pedestrian detection and prediction system (100)' according to the present invention (hereinafter referred to as the system (100)) is an integrated safety system that collects environmental information outside the vehicle through multiple sensors to prevent collision accidents with pedestrians or two-wheeled vehicles (including PMs) that may occur in the surroundings while driving, calculates the risk level in real time by precisely fusing and interpreting the information, and simultaneously provides visual, auditory, and tactile warnings to the driver and pedestrians outside the vehicle.

[0039] The system (100) of the present invention can be applied on the same principle regardless of the type of vehicle (passenger car, bus, large truck, special vehicle, etc.) and has a basic structure of continuously monitoring the entire 360° surrounding space including the front, rear, left, right, and, if necessary, the roof top of the vehicle. The system (100) is largely composed of components such as a sensor module unit (110), a multimodal fusion unit (120), an AI-based recognition and prediction unit (130), a vehicle integrated terminal (140), and an alarm display unit (150), and each component is closely linked to process data stepwise and hierarchically. This forms an organic structure capable of performing complex judgments in real time that a single component cannot perform.

[0040] The details of each component are as follows.

[0041] (1) Sensor module (110)

[0042] The sensor module (110) of the present invention is positioned in four directions, such as the front, rear, left, and right sides of the vehicle, and is configured to monitor a 360° omnidirectional area around the vehicle.

[0043] The sensor module (110) above is composed of a plurality of sensors having different detection principles—namely, an RGB camera (including a fisheye lens), an IR (infrared) camera, a multi-channel camera, and a multi-channel ultrasonic sensor, and an expensive LiDAR sensor may be optionally added for precise detection.

[0044] These sensors possess distinct physical characteristics and detection ranges, and are configured to maintain stable recognition performance in various environments, such as day and night and adverse weather conditions, by complementing each other's limitations.

[0045] RGB cameras acquire images in the visible light range and recognize visual patterns such as pedestrians, vehicles, lanes, signs, and obstacles around the vehicle. In particular, in this invention, a fisheye lens with a wide field of view of 180° or more is applied to each camera, so that the entire surroundings of the vehicle can be covered with only four cameras: front, rear, left, and right.

[0046] The camera's image is distorted in real-time by an image correction engine within the GPU, and exposure and color balance are automatically adjusted to minimize brightness differences between adjacent images. These corrected images are connected through a blending algorithm that optimizes the seam area and synthesized into a single panoramic image (Surround View Map, SVM). The advantages of RGB cameras lie in their high resolution and rich color and texture information. During the day, accurate object classification and lane recognition are possible by utilizing natural light. However, there is a limitation in that the signal-to-noise ratio (SNR) decreases due to reduced illumination and reflection in environments such as night, backlight, rain, or fog. To compensate for these drawbacks, the present invention applies HDR synthesis, a glare suppression filter, and environment-adaptive weighting adjustment together.

[0047] In addition, IR cameras detect the presence of people or animals based on the temperature difference (ΔT), which is the temperature difference between the heat (body temperature) emitted by a person or animal and the surrounding background (road, building, ground, etc.) in low-light or nighttime conditions. Since this is hardly affected by changes in illumination, pedestrian detection is possible even in dark environments that RGB cameras cannot recognize. For example, it remains inactive during the day and automatically activates when the brightness value detected by the illumination sensor falls below a preset value (e.g., 30 lux). The advantage of an IR camera is that it can detect the thermal shape of an object regardless of lighting conditions. On the other hand, it has the disadvantage that detection performance deteriorates in environments with high background temperatures or low thermal contrast, such as glass reflections. To overcome these limitations, the present invention combines the output signal of the IR camera with distance information from an ultrasonic sensor and a lidar sensor, and further applies a temperature-based thermal contrast correction algorithm to improve accuracy.

[0049] Additionally, multi-channel cameras (or stereo cameras) may be installed to improve recognition precision in specific areas (e.g., long distances ahead, intersection corners). By analyzing the parallax between the left and right images obtained through the stereo structure to calculate depth information (Depth Map), the distance estimation limitations of a monocular camera are compensated for. Furthermore, by observing from multiple viewpoints, areas of occlusion caused by objects can be re-observed from different angles. However, since the accuracy of a stereo camera largely depends on the geometric correction accuracy between cameras, the present invention ensures long-term stability by applying a temperature correction routine and a self-calibration algorithm.

[0050] In addition, multi-channel ultrasonic sensors are arranged on the underside of the vehicle to detect obstacles and pedestrians in a short-range area (approximately 0.2 to 3.0 m). More than 16 ultrasonic transducers are evenly distributed around the vehicle to transmit and receive sound waves in different directions. Distance is calculated by analyzing the phase difference (Δφ) and amplitude (A) from the reflected signals of the transmitted sound waves, and the direction of obstacles can also be estimated through the analysis of the time difference between multiple channels. The ultrasonic sensors are unaffected by weather, lighting, or color, and possess very high precision at short ranges. They are particularly effective for detecting low-lying objects such as pedestrian legs, pillars, and walls.

[0051] However, there are limitations such as low angular resolution due to the wide beam width and low reflectivity toward sound-absorbing material surfaces or high-speed moving objects. To compensate for these limitations, the present invention performs multi-channel array, beam pattern correction, and outlier removal filtering, and automatically corrects the reliability of the ultrasonic sensor using weights through cross-agreement with other sensors.

[0052] In addition, LiDAR sensors directly measure distance data using laser pulses and generate precise 3D point clouds in the medium to long distance (3 to 100 m) range. Through this, the height, width, and depth of objects around the vehicle can be quantitatively calculated. Since LiDAR sensors operate regardless of lighting conditions, they are stable even at night or in backlight. Advantages include the ability to obtain precise distance information and identify three-dimensional shapes. Disadvantages include the possibility of distance measurement errors occurring due to the attenuation or scattering of return signals in particle media environments such as precipitation or fog. Accordingly, the present invention evaluates return intensity and point density in real time, adjusts weights according to environmental conditions, and utilizes information from other sensors (e.g., IR, ultrasound) as an auxiliary tool.

[0053] Furthermore, the fisheye lens camera enables full-directional visibility around the vehicle using only four cameras (front, rear, left, and right), and serves as a reference plane for projecting distance information (ultrasonic and lidar) obtained from other sensors. Although fisheye images inherently exhibit distortion due to lens characteristics, this is internally corrected through Loop Uptable (LUT)-based back-projection and a mixed planar / cylindrical projection model. Additionally, vision optimization is performed by balancing brightness and color between adjacent camera images, providing natural and continuous 360° panoramic video.

[0054] (2) Multimodal fusion section (120)

[0055] The multimodal fusion unit (120) of the present invention is a module that performs time and space alignment to enable data obtained from different types of sensors to be integrated in a single time point and coordinate system.

[0056] This module aligns raw data collected specifically from RGB cameras, IR (infrared) cameras, multi-channel ultrasonic sensors, LiDAR sensors, etc., to a reference time (ts).

[0057] First, a common reference time is assigned to the data of all sensors. Since the signal transmission delay time or sampling period differs for each sensor, the fusion unit uses a time correction algorithm to unify the observation time of each sensor. For example, camera images may operate at 30fps, ultrasound at 10Hz, and lidar at 20Hz; however, in the present invention, the data from each sensor is interpolated or extrapolated to the nearest time to synchronize with the same reference time (ts).

[0058] At this time, by using information such as the vehicle's speed, steering angle, and acceleration to correct position errors (so-called egomotion) that occur while the sensors are moving, alignment to the same coordinate system is achieved not only on the time axis but also on the spatial axis. The data synchronized in this way is converted into spatial coordinates relative to the vehicle and spatially registered to the camera projection plane or panoramic coordinates (Surround View Map). In other words, distance information obtained from LiDAR or ultrasound is mapped to each pixel position (p) of the camera image, so that all sensors look at a point in the same space. Through this, differences in resolution or field of view between sensors can be corrected.

[0059] Next, the multimodal fusion unit calculates sensor-specific reliability weight maps to combine the aligned data at the pixel level. The reliability of each sensor is not simply given as a fixed ratio, but rather the frame quality (q) that determines the sharpness of the sensor image. s ), SNR (Signal-to-Noise Ratio), illuminance, saturation, rainfall status, and visibility (v) which determines whether the pixel is included in the field of view. s ), a probability value indicating whether it is not obscured by the vehicle body or other objects, and measurement reliability (c) determined by the intensity of the sensor data itself or the probability of feature detection (e.g., the confidence score of a deep learning feature map). s ), time series consistency (m) determined by whether the amount of change compared to the previous frame is stable s ), outlier suppression (o), a factor that excludes transient errors based on cross-agreement among multiple sensors s It is determined by considering various factors such as ).

[0060] The weight of each sensor is calculated by multiplying the above items, and

[0061]

[0062] It is defined as.

[0063] In the above formula represents the weight of the s-th sensor for the pixel at position p in the frame of time t.

[0064] In the above formula This is a predetermined basic ratio for each sensor based on environmental conditions such as day, night, and rain. For example, the weight of the infrared (IR) camera is set high at night, and the weight of the RGB camera is set high during the day.

[0065] The weights of all sensors are normalized, and the final fusion result for a pixel location (p) is calculated as follows.

[0066]

[0067] That is, the feature value (F^(s)) extracted from each sensor is used as a confidence weight ( The final integrated feature (F) is calculated by weighting the sum with ).

[0068] The above formula represents the characteristic values ​​of multiple sensors(s) corresponding to the same time t and the same spatial position p. reliability-based normalized weights for each sensor The final integrated feature map obtained by multiplying and summing This means generating [it]. As a core operation for implementing pixel-level multimodal data fusion, it is configured so that sensor contributions are dynamically adjusted by reflecting the quality, visibility, and time-series stability of each sensor.

[0069] for example,

[0070] At time p=(100,40), t=0.5s

[0071]

[0073] If so,

[0074] F=0.5*0.9+0.3*0.7+0.1*0.3+0.1*0.8=0.77

[0075] In this way, an integrated feature value is created in which the highly reliable sensor value is reflected more significantly.

[0076] The integrated feature map generated in this way refers to spatial feature representation data created by aligning different types of raw data or preprocessed feature values ​​obtained from multiple sensors to a common reference time point (t) and a vehicle reference coordinate system (p), and then combining them by weight based on the reliability, visibility, and quality indices of each sensor. The integrated feature map integrates information of multiple modalities, including sensor-specific image data, distance information, thermal information, and reflection intensity information, into pixel or spatial grid units. It is used as input for subsequent recognition algorithms such as object detection, tracking, and behavior prediction, and can be implemented as high-precision feature data that represents the vehicle's surrounding environment in a temporally continuous and spatially consistent manner. Furthermore, the integrated feature map can be combined with an intermediate feature representation generated by a deep learning-based feature extraction unit, or can be implemented as an expanded form of feature map by additionally fusing with vehicle dynamic information and road infrastructure (I2V) information.

[0077] In addition, the integrated feature map is subsequently transmitted to an AI-based recognition and prediction unit (130) and used for object recognition and collision risk prediction. Furthermore, the fusion unit of the present invention applies a boost rule that adjusts weights in real time according to environmental conditions. For example, the weight of the IR camera is increased by +20% at night, the weight of the ultrasonic sensor is increased by +30% when rain or snow is detected, and the weight of the RGB camera is temporarily lowered when backlight or glare is detected.

[0078] In addition, when a non-line-of-sight situation is detected where a pedestrian is obscured by an object in the forward field of view while the vehicle is turning right, the weights of the non-line-of-sight sensors (LiDAR, ultrasound) are automatically increased to strengthen surveillance of that area. Finally, the multimodal fusion unit applies an Empowered Mode (EMA) algorithm to mitigate abrupt changes in weights. This prevents the alarm from flickering due to instantaneous signal noise or disturbances between sensors, and provides more stable fusion results.

[0079] Through this configuration, the multimodal fusion unit of the present invention maximizes the advantages of each sensor while compensating for their disadvantages, enabling stable 360° recognition of the surrounding environment even in day and night, adverse weather conditions, or shielded situations. This has the effect of significantly improving pedestrian detection accuracy, distance estimation precision, and the reliability of collision risk prediction compared to conventional single-sensor-based surveillance systems or simple image fusion techniques.

[0080] (3) AI-based recognition and prediction unit (130)

[0081] The AI-based recognition and prediction unit (130) of the present invention receives an integrated feature map output from a multimodal fusion unit (120) as input, and sequentially performs an object recognition step (S1), an object tracking step (S2), an intention prediction step (S3), and a collision risk prediction step (S4) as shown in FIG. 3, thereby detecting objects such as pedestrians, bicycles, motorcycles, and other vehicles around the vehicle, and performing the function of calculating their movement intentions and collision risk with the vehicle in real time.

[0082] The AI-based recognition and prediction unit (130) analyzes spatial and time-series information represented by the integrated feature map through a deep neural network structure and functions as a high-level intelligent processing unit that not only finds "what is where" but also determines "how the object will move in the future and the likelihood of it colliding with a vehicle as a result."

[0083] First, the object recognition step (S1) is performed by a deep learning-based CNN (Convolutional Neural Network). The CNN has a structure that hierarchically extracts features ranging from basic features such as low-dimensional edges and corners to high-dimensional semantic features such as pedestrian shapes, vehicle contours, and bicycle silhouettes by repeatedly applying multiple stages of convolution (conv) operations, non-linear activation, and pooling operations to an input image or integrated feature map. In this embodiment, a lightweight YOLOv8-tiny model is adopted to account for computational load and power consumption, and by running it on a vehicle-mounted FPGA accelerator, a processing speed of more than 30 frames per second (30 FPS) is secured for an input of 640×640 pixels. This processing performance is sufficient to continuously recognize surrounding objects without missing them, even when the vehicle is traveling at speeds of tens of km / h.

[0084] YOLO-based models have a one-stage structure that divides the entire input into small grid cells and simultaneously predicts for each cell whether a specific object exists at that location and, if so, what its bounding box and class should be determined. Specifically, in the output layer of the model, for each candidate location, bounding box parameters such as center coordinates, width, and height, scores for each class (pedestrian, bicycle, vehicle, etc.), and a Confidence Score indicating the presence of the object are calculated. Subsequently, in the post-processing stage, an algorithm such as Non-Maximum Suppression (NMS) is applied to remove the rest of the multiple overlapping candidate boxes while retaining only those with high confidence, thereby ultimately selecting only a small number of non-overlapping object candidates.

[0085] The results processed in this manner are organized and output for each object in the form of a “Bounding Box (location and size in the image)”, a “Class ID (identifier indicating the type of object, such as a pedestrian, bicycle, vehicle, or motorcycle)”, and a “Confidence Score (confidence or probability value of recognizing the object)”. Here, the Confidence Score is expressed as a value between 0 and 1, and a higher value indicates a higher probability that the object actually exists. Based on a preset threshold (e.g., 0.3 or 0.5), the system removes objects with low confidence levels below the threshold as noise and transmits only objects with confidence levels above the threshold to the subsequent object tracking step (S2).

[0086] In addition, the AI-based recognition and prediction unit (130) is designed so that it does not simply apply CNN only to RGB images, but can use the integrated feature map generated by the multimodal fusion unit (120) as input. For example, if RGB channels, IR thermal image channels, LiDAR distance channels, and ultrasound-based near-field channels are provided in a multi-channel form within the integrated feature map, the input to the CNN accepts this multi-channel data as is, thereby ensuring stable object recognition performance even under adverse conditions, low light, or rain. That is, the multiple sensor information included in the integrated feature map is processed together in the initial layer of the CNN, and through this, much richer features are extracted than those based on a single camera.

[0087] In addition, the object tracking step (S2) is a step for connecting objects such as pedestrians, bicycles, and vehicles, which are individually detected in each frame in the preceding object recognition step (S1), along the time axis to track them as the same object. In other words, if the S1 step is a process of determining “what is where in this frame screen?”, the S2 step can be described as a process of determining “whether this object just recognized is the same as an object recognized in the previous frame and where it will move in the future.” If this step is not performed properly, a pedestrian may be misidentified as a new object in a situation where they are briefly obscured and then reappear, or the object's movement path may be broken, leading to inaccurate prediction of intent and judgment of collision risk; therefore, a stable time-series tracking algorithm is essential.

[0088] To this end, the present invention applies a time series tracking network based on BiLSTM (Bidirectional Long Short-Term Memory). LSTM is a type of recurrent neural network (RNN) specialized for processing time series data, and it is a structure that solves the vanishing gradient problem occurring in simple RNNs by controlling whether to remember or forget information from past points in time for a long period through an internal gate structure. In the present invention, by using a BiLSTM that extends this LSTM into a “bidirectional” structure, it obtains more precise tracking results by simultaneously considering not only information in one direction (past → present) but also information in the opposite direction (future → present).

[0089] Specifically, the input to the object tracking step (S2) consists of dynamic information, such as the position, velocity, direction of movement, and acceleration of each object transmitted from the object recognition step (S1) at each time t. The position can be expressed, for example, as the center coordinates and bounding box size in a panoramic coordinate system or an image coordinate system, and the velocity and acceleration are calculated based on the change in position between consecutive frames. The direction of movement is calculated from the directional component of the velocity vector, and this information is provided as input to the BiLSTM network in the form of a time-series vector.

[0090] The BiLSTM network simultaneously processes features of the same object candidate at time points t-1, t, and t+1 in bidirectional LSTM cells. One LSTM learns movement patterns accumulated from past time points (t-1, t-2, …), while the other LSTM considers the continuity of movement that may be observed at future time points (t+1, t+2, …) to more reliably infer how the object is moving at the current time point t. By using this bidirectional structure, it helps maintain the object ID without interruption based on the flow of the entire sequence, even when a pedestrian is briefly obscured by another vehicle and reappears, or when the bounding box shakes due to temporarily low recognition confidence.

[0091] Furthermore, in real road environments, instantaneous prediction errors may occur using only BiLSTM due to sensor noise, image shake, and temporary occlusion. To compensate for this, the present invention incorporates a Kalman filter as an auxiliary component. The Kalman filter is a stochastic filtering technique that combines current observations with a prediction model to provide an estimate with minimized noise; it is particularly suitable for predicting the short-term position and speed of pedestrians and vehicles moving in straight lines or gentle curves.

[0092] In this embodiment, the Kalman filter operates by using the predicted position and velocity of the next time step calculated by the BiLSTM network as the predicted value, and the position and velocity observed by the actual sensor as the observed value, and performing a weighted sum of the two values. If the difference between the predicted value and the observed value is large, the difference is determined to be noise or a momentary recognition error, and the filtered result is gradually adjusted so that the movement trajectory of the object between consecutive frames is smooth and natural.

[0093] Through the object tracking step (S2) configured in this manner, the present invention assigns a unique tracking ID to each object around the vehicle and can stably maintain a continuous movement trajectory over time. As a result, in the subsequent intention prediction step (S3), the pedestrian's crossing intention can be determined based on the pattern of changes in trajectory, speed, and direction accumulated over a certain period of time, rather than sporadic location information, and in the collision risk prediction step (S4), a more reliable risk level can be calculated by combining this tracking information and the TTC. Therefore, the object tracking step (S2) can be considered a core time-series analysis step that goes beyond simple location interpolation and determines the reliability of the entire safety system of the present invention.

[0094] In addition, the intention prediction step (S3) is a step that probabilistically determines (probability value P_cross) whether the pedestrian is actually intending to cross the road based on time-series information about the pedestrian derived from the preceding object recognition step (S1) and object tracking step (S2). In other words, since the collision risk cannot be accurately determined based solely on the fact that “a pedestrian is standing near the road,” the intention prediction step (S3) performs the function of distinguishing “whether this pedestrian will soon enter the road or remain” by analyzing various behavioral and situational factors over time, such as the pedestrian’s head direction, body rotation, movement speed, relative distance from the vehicle, and the density of surrounding pedestrians.

[0095] To this end, the intention prediction step (S3) first receives time-series characteristic values ​​for each pedestrian provided in the object tracking step (S2) as input. Specifically, for each pedestrian, information such as (i) head orientation, (ii) body rotation angle, (iii) moving speed v(t), (iv) acceleration, (v) relative angle with respect to the vehicle's direction of travel, (vi) relative distance d(t) with respect to the vehicle, (vii) the location of the sidewalk / roadway boundary where the pedestrian is located, and (viii) the distribution and density of surrounding pedestrians, measured over consecutive frames from time tk to t, is organized in the form of a time-series vector. The head orientation is calculated through a pose estimation algorithm or keypoint detection for the pedestrian's face or head region, and the body rotation angle is calculated based on the direction of the shoulder / pelvis or the major axis direction of the Bounding Box. Movement speed and acceleration can be easily obtained using the change in position already calculated during the object tracking phase, and the relative distance and relative angle with respect to the vehicle are calculated by projecting the pedestrian's position onto the vehicle's reference coordinate system.

[0096] It is important to note that these multiple features are not static values ​​at a fixed point in time, but time-series data that change over a certain time interval. For example, if a pedestrian initially stands on the roadside, then turns their head toward the road, rotates their body toward the road, and then begins to walk toward the inside of the road, this behavior appears as a “sequential change pattern.” Therefore, in the intention prediction step (S3) of the present invention, instead of simply looking at the value of the current frame, changes in consecutive frames corresponding to a certain time interval (e.g., the last 1 to 2 seconds) are input as a sequence, and the pedestrian's crossing intention is inferred using a deep learning-based time-series model (e.g., LSTM, BiLSTM, GRU-based network, or 1D CNN-based time-series classifier) ​​that has learned the pattern of this sequence.

[0097] In the training phase, the intention prediction model learns from actual road footage or simulation data with two (or more multi-stage) labels, such as “attempted crossing” and “no intention to cross.” For example, in the training data, a pattern in which a pedestrian stands at the edge of the road for a certain period of time with their head and body turned away from the direction of vehicle movement and moving at a speed of nearly zero can be labeled as “no intention to cross,” while conversely, a pattern in which a pedestrian turns their head toward the road multiple times to look around (resulting in a large change in head direction Δθ), rotates their torso toward the road, and moves toward the inside of the roadway with a speed v of 0.5 m / s or more is labeled as “intention to cross,” and the model learns this.

[0098] In the present invention, the value calculated by the intention prediction step (S3) is a probability value P_cross representing the probability that a pedestrian will cross the road within a certain time. This value is expressed between 0 and 1, and the higher the value, the stronger the intention that “the pedestrian will cross the road in the future.” As in the example described in the specification, if the change in head direction Δθ exceeds 30°, the movement speed v exceeds 0.5 m / s, and the relative distance d from the vehicle decreases to less than 10 m, the intention prediction model recognizes this pattern as a high-risk behavior pattern according to learned parameters, and P_cross can be calculated as 0.7 or higher (i.e., 70% or higher). In this case, the system classifies the pedestrian as having “intention to cross” and reflects this as a high risk in the collision risk prediction step (S4), which calculates a risk index (RR) by combining it with TTC (Time-To-Collision). Conversely, if a pedestrian is standing on the roadside but their body is facing inward toward the sidewalk, there is little change in head direction, their movement speed is minimal at 0.2 m / s or less, and their position is significantly far from the crosswalk area, the model determines this pattern as "waiting or stopped" behavior and calculates a low P_cross, even though the absolute distance from the vehicle is relatively close. In this case, even if the TTC is short in the collision risk prediction step (S4), the final risk index is evaluated as relatively low because the intentional probability is low, thereby reducing unnecessary excessive warnings.

[0099] In addition, the intention prediction step (S3) can consider not only individual pedestrians but also the distribution and collective behavior of surrounding pedestrians. For example, if a pedestrian is loitering at the edge of the road and other pedestrians located in the same direction have already started crossing, it can be considered that the likelihood of that pedestrian crossing increases in the environmental context. To this end, the input to the intention prediction model may include collective characteristics such as the location, direction of movement, and density of surrounding pedestrians, and the model synthesizes this information to produce a more sophisticated P_cross.

[0100] In conclusion, unlike conventional ADAS based solely on simple location information, the intention prediction step (S3) of the present invention is a core higher-level recognition step that enables the calculation of a more realistic and reliable risk level in the subsequent collision risk prediction step (S4) by probabilistically determining whether “the pedestrian actually has the intention to cross the road” through time-series analysis of various behavioral and situational variables, such as the direction of the pedestrian’s head and body, changes in speed, relative distance, and surrounding pedestrian conditions, unlike conventional ADAS.

[0101] In addition, the collision risk prediction step (S4) of the present invention is a step that synthesizes the information calculated in the preceding object tracking step (S2) and intention prediction step (S3) to calculate the probability of an actual collision occurring between a vehicle and a pedestrian (or bicycle, motorcycle, etc.) as a quantitative indicator, and finally determines whether to trigger an alarm based on the result. In other words, if the object recognition step (S1) and the object tracking step (S2) provide information on “who is currently nearby and where they are moving,” and the intention prediction step (S3) provides information on “who among them is most likely to cross the road,” the collision risk prediction step (S4) performs the role of finally determining “how much risk there is of this vehicle actually colliding with this pedestrian, and whether an alarm should be sounded now” based on all this information.

[0102] First, in the collision risk prediction step (S4), Time-To-Collision (TTC) is calculated as a basic physical quantity. TTC is a value defined based on the relative distance d and relative speed v_rel between a vehicle and a specific object (e.g., a pedestrian), and reflects how quickly the two objects are approaching each other per unit of time. Generally, TTC is expressed as follows.

[0103] TTC = d / v_rel

[0104] Here, d represents the straight-line distance (in meters) from the vehicle-based coordinate system to the pedestrian, and v_rel represents the relative speed (in meters / second) based on the vehicle's direction component. For example, if a vehicle is traveling straight at 50 km / h and a pedestrian suddenly enters the road in the direction of the vehicle's movement, causing the distance between them to rapidly decrease, v_rel will increase to a positive value, and accordingly, the TTC can be calculated as a very short value such as 2 seconds or 1.5 seconds. Conversely, if the pedestrian moves away from the vehicle or the vehicle is decelerating, the v_rel value will decrease or become negative, and the TTC may become very long or interpreted as virtually no collision occurring. In the present invention, limited to situations where an actual collision may occur, if v_rel is 0 or negative, the TTC is considered to be infinite or a non-hazardous state and is excluded from subsequent calculations.

[0105] However, it is difficult to sufficiently explain the risk of collision based on TTC alone. For example, even if the TTC is calculated to be 3 seconds as the distance between the pedestrian and the vehicle gradually decreases, if the pedestrian is simply standing at the edge of the sidewalk looking only at their mobile phone, the likelihood of actually crossing the road may be low. To compensate for this limitation, the present invention calculates a final risk index by combining the pedestrian's crossing intention probability P_cross, calculated in the intention prediction step (S3), with the TTC. P_cross represents the probability that the pedestrian will enter the inner side of the road or attempt to cross within a certain period of time in the future, and is derived as a result of analyzing time series patterns such as changes in head direction, body rotation, movement speed, relative position with the vehicle, and the distribution of surrounding pedestrians.

[0106] In the collision risk prediction step (S4), these two values, namely TTC and P_cross, are combined to calculate the risk index (RR) as follows.

[0107] RR = P_cross / TTC

[0108] As can be seen from the equation above, the shorter the TTC and the larger the P_cross, the greater the RR value increases. This is a mathematical expression of the intuitive judgment that “the higher the likelihood that a pedestrian will actually cross the road and the shorter the time remaining until a collision with a vehicle, the higher the risk.” For example, if P_cross is 0.7 and the TTC is 2.0 seconds, the RR is 0.35, but if the TTC becomes shorter to 1.0 seconds, the RR increases to 0.7, significantly increasing the risk. Conversely, even if the TTC is very short, if P_cross is estimated to be low, such as 0.1 or less, the RR is not large, which can reduce unnecessary emergency alerts.

[0109] In order to operate as a practical driver warning system, the present invention sets certain threshold conditions based on the calculated RR value and TTC value. As an example, the case where RR ≥ 0.5 or TTC ≤ 2.0 seconds can be defined as a “high-risk situation requiring immediate response.” The condition RR ≥ 0.5 refers to a case where the pedestrian’s crossing intention is significantly high (e.g., P_cross is 0.7 or higher) and the TTC is relatively short, while the condition TTC ≤ 2.0 seconds encompasses a case where emergency braking or warning is required because the time remaining until a physical collision is very short, even if the intention prediction is somewhat uncertain. If either of these two conditions is satisfied, the collision risk prediction step (S4) determines the situation as high-risk and immediately transmits the result to the vehicle integrated terminal (140).

[0110] When the vehicle integrated terminal (140) receives this warning signal, it considers additional information such as the current vehicle speed, brake clearance, road surface condition (dry road surface, wet road surface, etc.), and road speed limit, and decides whether to stop at a simple warning (HUD display, warning sound output) or to activate the automatic emergency braking (AEB) system. For example, in a vehicle traveling at high speed, if the TTC is calculated to be 1.5 seconds and the RR is 0.8, the driver's reaction time may be insufficient with only a simple warning, so the system may operate by applying a certain level of automatic braking force in advance while simultaneously giving a strong warning to the driver.

[0111] As such, the collision risk prediction step (S4) can be described as a core algorithm step of the present invention that, unlike conventional methods that simply mechanically calculate distance and speed, performs a risk judgment closer to the actual road situation through a risk index (RR) combining the pedestrian's behavioral intention (P_cross) and TTC, thereby reducing unnecessary false alarms and enabling rapid and strong warnings in truly dangerous situations.

[0112] 5) On-device AI computation structure

[0113] The AI-based recognition and prediction unit of the present invention includes an on-device processor based on an FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit) to perform object recognition, object tracking, intent prediction, and collision risk determination in real time inside a vehicle. The on-device processor is configured in a parallel structure with the vehicle's general CPU, whereby the CPU is responsible for management functions such as sensor data collection, scheduling, interfacing with the vehicle ECU, and alarm control, while the FPGA or ASIC is responsible for high-load computations such as CNN-based object recognition, BiLSTM-based time series analysis, Time-To-Collision (TTC) calculation, and risk assessment.

[0114] In particular, the on-device AI structure of the present invention is designed so that the CNN stage, BiLSTM stage, and risk assessment stage are connected via a streaming pipeline, allowing the stages to overlap and proceed in parallel rather than waiting for the computation of each module to be completed sequentially. Accordingly, the total frame processing latency is not calculated by accumulating the computation time of each stage, but is limited to the processing time of the longest stage among the modules constituting the pipeline, thereby maintaining the overall system latency at 50 ms or less. This latency provides sufficient processing performance to determine object behavior changes and collision risks in real time, even when the vehicle is traveling at high speeds.

[0115] In addition, the deep learning model executed within the FPGA does not use the floating-point (FP32) based model as is, but is converted into an INT8 integer-based quantization model to maximize computational efficiency. The INT8 quantization model significantly reduces computational load and memory usage, thereby improving power efficiency. In this invention, it is preferable to configure the entire on-device AI computing unit to limit power consumption to 20 W or less. This power efficiency enables stable AI computation even in commercial vehicles or electric vehicles requiring long-duration driving, and has the advantage of minimizing heat generation and system load.

[0116] According to the above-described on-device AI structure, the vehicle can independently perform real-time decisions without connecting to an external cloud server or remote AI infrastructure. In other words, object recognition and collision risk prediction functions are performed normally even in environments where the vehicle's communication network is unstable, such as tunnels, underground parking lots, and communication blind spots; furthermore, since video and sensor data are not transmitted externally, it is advantageous in terms of personal information protection. Accordingly, the on-device processor of the present invention enables the entire system to operate as a complete edge AI-based structure and functions as a core component that enables real-time analysis of the vehicle's surrounding environment and immediate risk response.

[0117] (4) Vehicle integrated terminal (140)

[0118] The vehicle integrated terminal (140) according to the present invention is a core interface device that enables the driver or the vehicle control system to respond immediately by linking analysis values ​​such as object recognition results, tracking information, intention probability, and collision risk calculated by the AI-based recognition and prediction unit (130) with the vehicle's own control unit (ECU). That is, the vehicle integrated terminal (140) serves as a medium that enables all recognition and prediction functions of the system to lead to actual vehicle operation, and can be described as a central integrated processing unit that comprehensively manages the vehicle internal communication network, external infrastructure communication, alarm output, and emergency braking control.

[0119] First, the vehicle integrated terminal (140) performs the function of collecting and synchronizing vehicle dynamic information from the vehicle's internal communication network. Multiple communication protocols such as CAN (Controller Area Network), LIN (Local Interconnect Network), and Automotive Ethernet are used within the vehicle, and the integrated terminal (140) periodically acquires various vehicle status information, such as vehicle speed, steering angle, braking pressure, engine rotational speed (RPM), and the ON / OFF status of the brake switch, through these communication networks. The collected vehicle dynamic data is used as an important auxiliary variable when calculating Time-To-Collision (TTC) and risk level by the AI-based recognition and prediction unit (130). For example, since vehicle speed and braking pressure are essential factors for determining the possibility of collision and avoidance, the integrated terminal (140) plays a role in significantly improving prediction accuracy by aligning the sensor recognition results with the vehicle operation information in time.

[0120] In addition, the vehicle integrated terminal (140) includes an Infrastructure-to-Vehicle (I2V) communication function to receive various traffic information provided by external traffic infrastructure. Specifically, it receives external data in real time, such as SPaT (Signal Phase and Timing) information transmitted from intersection traffic lights, the presence of pedestrians detected by pedestrian detection sensors or roadside cameras, and construction zone guidance information, and reflects this in the analysis of the situation around the vehicle. The I2V communication function effectively compensates for the visibility limitations of existing vehicle-only sensor-based systems by enabling the detection of pedestrian presence through road infrastructure sensor information even when the front of the vehicle is obscured by buses, trucks, buildings, etc., resulting in a Non-Line-Of-Sight (NLOS) situation.

[0121] Additionally, the vehicle integrated terminal (140) includes a function to determine a risk level based on the risk level (RR) or TTC value transmitted from the AI-based recognition and prediction unit (130) and to output an appropriate warning signal to the warning display unit (150). In the present invention, the risk level can be divided into three stages, for example, 'caution', 'warning', and 'urgent'. It can operate by generating an immediate 'urgent warning' when the risk index RR is 0.8 or higher, and generating a 'caution warning' when the RR is 0.5 or higher but less than 0.8. Such warning signals can be transmitted to the driver through a HUD (Head-Up Display), cluster warning light, warning sound, steering wheel vibration, etc., and to pedestrians outside the vehicle through a front LED panel, warning light, speaker, etc.

[0122] Additionally, the vehicle integrated terminal (140) can perform the function of controlling Automatic Emergency Braking (AEB) in conjunction with the vehicle ADAS function when it is determined that the vehicle has a high risk of collision. For example, if the TTC is 2.0 seconds or less or the RR rises above a threshold, the integrated terminal (140) can control the vehicle to automatically decelerate or stop by transmitting a braking command to the ECU and the brake control device. Such a function is useful for preventing accidents caused by a driver's reaction delay and plays a role in minimizing the risk of collision in various driving situations, such as highways, city roads, and intersections.

[0123] In short, the vehicle integrated terminal (140) integrates internal vehicle sensor and ECU information and external infrastructure information, comprehensively determines the risk of collision based on AI prediction results, and functions as a comprehensive control device that connects to alarm output and vehicle control. This is a key component that enables the present invention to operate as an intelligent vehicle safety platform for ensuring driver safety in actual driving environments, going beyond a simple object recognition system.

[0125] (5) Alarm display unit (150)

[0126] The alarm display unit (150) according to the present invention is composed of a Human-Machine Interface (HMI) device for intuitively and quickly conveying a dangerous situation to the driver and pedestrians outside the vehicle according to the danger level calculated by the AI-based recognition and prediction unit (130) and the vehicle integrated terminal (140). That is, the alarm display unit (150) performs the role of enabling the driver and pedestrians to simultaneously perceive danger and take appropriate avoidance actions in a situation where a collision is possible by integrally controlling visual, auditory, and tactile output means placed inside and outside the vehicle.

[0127] First, the driver warning output unit includes various HMI devices installed inside the vehicle, such as a Head-Up Display (HUD), an instrument cluster display, a warning speaker, a steering wheel vibration motor, and a seatbelt tensioner. The vehicle integrated terminal (140) determines the danger level into three stages—caution, warning, and emergency—based on AI prediction results, and the warning display unit (150) provides the driver with visual, auditory, and tactile warnings corresponding to the respective stages. For example, in the caution stage, a warning message is displayed along with a yellow icon on the HUD or cluster screen, and in the warning stage, an orange warning light flashes and a medium-tone warning sound is output. In the emergency stage, a high-pitched beep or voice warning is output simultaneously with a red warning display, and additional tactile warnings are provided by causing vibration in the steering wheel and temporarily tightening the seatbelt tensioner. This multi-warning method has the effect of minimizing reaction delay by enabling the driver to immediately detect danger even when they cannot visually check the screen.

[0128] In addition, the pedestrian warning output unit includes an LED warning panel, a flashing light, and an external speaker (soundbar) positioned at the front of the vehicle. The pedestrian warning plays a crucial role, particularly at night, in inclement weather, or in urban environments where visibility is poor, enabling pedestrians to detect danger and take evasive action independently of the driver's awareness of the warning.

[0129] In the embodiment, text or pictogram messages such as "Vehicle Approaching" or "Caution" are displayed via the front LED panel of the vehicle, and in emergency situations, a red flash rapidly blinks to attract visual attention. During the day, blue or high-intensity LEDs are used due to high external light levels, while at night, red or orange flashing lights are utilized to maximize the visibility of the warning indication. External speakers emit voice guidance or warning sounds such as "Caution: Incoming Vehicle" or "Vehicle Approaching," thereby effectively conveying dangerous situations to the visually impaired or pedestrians who have difficulty perceiving danger based solely on visual information.

[0130] The warning display unit (150) configured in this manner is designed so that drivers and pedestrians can perceive danger independently or simultaneously, and plays an important role in preventing the possibility of accidents in advance in various environments such as road intersections, school zones, and sidewalk-roadway boundaries. That is, the warning display unit (150) of the present invention is not merely a device that outputs a warning sound, but functions as a vehicle-pedestrian integrated safety interface that enables the provision of multifaceted, intuitive, and immediate warnings regarding dangerous situations by linking the vehicle's internal HMI with an external warning device.

[0131] FIG. 3 is a flowchart illustrating the overall execution steps of the present invention.

[0132] The execution steps of the present invention consist of (A) a sensor data collection and preprocessing step, (B) a time-spatial alignment and integrated feature map generation step, (C) an object recognition step, (D) an object tracking step, (E) an intent prediction step, (F) a collision risk prediction and risk level calculation step, (G) an on-device AI computation execution step, (H) a vehicle integrated terminal linkage and vehicle status information fusion step, and (I) an alarm output and vehicle control step.

[0133] In the sensor data collection and preprocessing stage, multiple sensors are operated to observe the external environment of the vehicle. Specifically, RGB cameras (including fisheye lenses), IR (infrared) cameras, multi-channel cameras (possibly including stereo cameras), multi-channel ultrasonic sensors, and optionally LiDAR sensors are positioned on the front, rear, left and right sides of the vehicle and, if necessary, on the roof, and the sensors continuously collect image, distance, and thermal information over time for a 360° omnidirectional area around the vehicle.

[0134] In the above steps, the RGB camera acquires images in the visible light region, and the IR camera is automatically activated when the illuminance measured by the illuminance sensor drops below a preset threshold (e.g., 30 lux) to acquire thermal images based on body heat contrast (ΔT). Multichannel ultrasonic sensors are arranged around the perimeter of the lower part of the vehicle to acquire distance information in the short-range (e.g., 0.2 to 3 m) region, and the LiDAR sensor uses laser pulses to generate a three-dimensional point cloud in the medium-to-long-range (e.g., 3 to 100 m) region.

[0135] For the raw data acquired from each sensor, preprocessing operations such as lens distortion correction, color and brightness equalization, noise filtering of ultrasonic and lidar signals, and temperature correction of thermal images are performed. In particular, images from fisheye cameras undergo geometric correction using a back-projection LUT and color and brightness correction between adjacent cameras, after which they are blended to minimize the seam area and synthesized into a 360° panoramic image (Surround View Map). The sensor data preprocessed in this way is used as basic data for performing time-spatial alignment in subsequent steps.

[0136] In addition, during the time-space alignment and integrated feature map generation stages, alignment processing is performed to align the sensor data to a common reference time and a common coordinate system. First, since each sensor, such as cameras, ultrasound, and LiDAR, has different frame rates and sampling periods, the multimodal fusion unit sets a reference time (ts) and performs interpolation or extrapolation based on the timestamps of each sensor to align the data of all sensors to correspond to the reference time (ts). At this time, vehicle dynamic information such as vehicle speed, steering angle, and acceleration is reflected, and positional errors caused by movement of the sensor mounting location (ego motion) are corrected.

[0137] After time alignment is completed, each sensor data is converted to the vehicle reference coordinate system and mapped to the same pixel location (p) on the camera projection plane or the panoramic coordinate system. For example, LiDAR point cloud data and ultrasonic distance data are mapped to pixels in the camera image through coordinate transformation using extrinsic and intrinsic parameters, and are configured so that information of multiple modalities, such as RGB, column, distance, and reflection intensity, exists simultaneously for the spatial point represented by the same pixel.

[0138] Next, for each sensor, frame quality (q s ), Visibility(v s ), measurement reliability (c s ), time series consistency(m s ), outlier suppression(o s Reliability weights composed of ) etc. are calculated, and basic weights (α) per sensor are determined according to environmental conditions (day / night / rain / fog, NLOS status, etc.). s ) is dynamically adjusted. Subsequently, the final weight w for sensor s at pixel position p and time t. s Calculate (p,t) and normalize it

[0139]

[0140]

[0141] The final integrated feature map F(p,t) is generated by the mathematical formula.

[0142] The integrated feature map generated in this way is high-precision spatial feature data that integrates information such as images, distances, heat, and reflection intensity acquired from multiple sensors in a common viewpoint and coordinate system, and is subsequently provided as input to an AI-based recognition and prediction unit (130).

[0143] Additionally, in the object recognition step (C), objects around the vehicle are detected using a deep learning-based CNN (Convolutional Neural Network) that takes the integrated feature map as input. In this embodiment, the CNN is implemented with a YOLOv8-tiny structure and is configured to operate on an on-device processor (FPGA / ASIC) at a resolution of 640×640 pixels and a speed of 30 FPS or higher.

[0144] CNN repeatedly performs convolution, activation, and pooling operations on multi-channel inputs of integrated feature maps to hierarchically extract features ranging from low-dimensional features such as edges and corners to high-dimensional semantic features such as pedestrian shapes, bicycle silhouettes, and vehicle contours. In the output layer, the location and size of the bounding box, class ID (pedestrian, bicycle, two-wheeled vehicle, vehicle, etc.), and confidence score are calculated for each grid cell, and duplicate boxes are removed through Non-Maximum Suppression (NMS), so that meaningful objects are finally detected in each frame.

[0145] In addition, in the object tracking step (D), objects detected frame by frame in the object recognition step (C) are connected along the time axis and tracked as the same object. To this end, dynamic information such as position, velocity, direction of movement, and acceleration is extracted for each object frame by frame, and this time series data is input into a BiLSTM-based time series tracking network.

[0146] The BiLSTM network simultaneously considers information from past time points (t-1, t-2, …) and future time points (t+1, t+2, …) in both directions to calculate a continuous trajectory, ensuring that object IDs are not interrupted even if temporary occlusion or a decrease in recognition reliability occurs. Additionally, a Kalman filter is applied as an auxiliary component; by combining the position and velocity predicted by the BiLSTM with actual observations, it suppresses noise and instantaneous errors, providing smooth and natural object trajectories.

[0147] Next, in the intention prediction step (E), the time-series information for each pedestrian derived in the object tracking step (D) is used to calculate a probability value P_cross indicating whether each pedestrian has an intention to cross the road. To this end, time-series characteristics such as head direction, body rotation angle, movement speed, relative distance from the vehicle, relative angle with respect to the vehicle's direction of travel, sidewalk / roadway boundary location, and the distribution and density of surrounding pedestrians are extracted for each pedestrian.

[0148] The change patterns of these features consist of a sequence of specific time intervals (e.g., the last 1 to 2 seconds) and are input into a time series classification model based on LSTM, BiLSTM, GRU, or 1D CNN. During the training phase, the model is trained using data labeled as “attempted to cross” and “no intention to cross,” and learns the behavioral pattern in which a driver looks around, turns their body toward the road, and then enters the roadway at a speed exceeding a certain threshold as “intention to cross.” In the inference phase, the model analyzes the input sequence to calculate the probability of crossing intention P_cross for each pedestrian as a value in the range of 0 to 1.

[0149] Next, in the collision risk prediction step (F), the results of the object tracking step (D) and the intention prediction step (E) are combined to evaluate the collision risk between the vehicle and the pedestrian using a quantitative indicator. First, Time-To-Collision (TTC) = d / v_rel is calculated based on the relative distance d and relative speed v_rel in the vehicle reference coordinate system, and if v_rel is 0 or less, it is considered a situation where no collision occurs, and the TTC is treated as a non-hazardous state.

[0150] Next, the risk index RR is calculated by combining the above TTC and the intention prediction probability P_cross as RR = P_cross / TTC. Since RR increases as TTC is shorter and P_cross is larger, a high risk is assessed when the pedestrian has a high intention to actually cross the road and the time remaining until a collision with a vehicle is short. For example, if P_cross is 0.7 and TTC is 1.0 second, RR is calculated as 0.7, which can be classified as a high-risk situation.

[0151] In the present invention, a high-risk situation is determined when a preset threshold condition for RR and TTC, for example, RR ≥ 0.5 or TTC ≤ 2.0 seconds, and when the above condition is satisfied, a high-risk alarm signal can be output to the vehicle integrated terminal (140).

[0152] In addition, the on-device AI computation execution step (G), which is performed in the computation steps (B) to (F) above, is preferably executed on an FPGA or ASIC-based processor inside the vehicle through an on-device AI computation structure. In this step, CNN-based object recognition, BiLSTM-based time series tracking, intent prediction model, and TTC·RR calculation are implemented in a streaming pipeline structure so that the total system latency is maintained at 50 ms or less.

[0153] The on-device AI processor utilizes an INT8 quantization model to reduce computational load and memory usage, and limits power consumption to 20 W or less, thereby suppressing heat generation and power consumption even in long-duration driving environments and enabling stable real-time processing. Due to this on-device structure, the method of the present invention can perform real-time decisions independently on the vehicle without relying on an external cloud server.

[0154] Next, in the vehicle integrated terminal linkage step (H), the AI-based recognition and prediction results are transmitted to the vehicle integrated terminal (140), and the integrated terminal (140) combines vehicle dynamic data (speed, steering angle, braking pressure, engine RPM, brake ON / OFF status, etc.) collected through the vehicle internal communication network (CAN, LIN, Automotive Ethernet, etc.) and intersection signal, pedestrian detection infrastructure information, construction zone information, etc. received through road infrastructure (I2V) communication.

[0155] The vehicle integrated terminal (140) determines the final risk level as multiple stages such as 'caution', 'warning', and 'urgent' by considering the AI ​​prediction results (RR, TTC, P_cross, etc.), vehicle status information, and external infrastructure information together, and outputs a control signal to the alarm display unit (150) and ADAS control unit (AEB, etc.) according to the determination result.

[0156] Finally, in the warning output and vehicle control step (I), based on the judgment result of the vehicle integrated terminal (140), a warning is output to the driver and pedestrian through the warning display unit (150), and vehicle control is performed if necessary. The warning for the driver is provided through the HUD, cluster display, warning sound speaker, steering wheel vibration, seatbelt tensioner, etc., and is configured so that a stronger visual, auditory, or tactile warning is output as the risk level increases. The warning for pedestrians is provided through the LED warning panel on the front of the vehicle, a warning light, and an external speaker, and outputs messages or voices such as “Vehicle approaching” or “Caution: Incoming Vehicle” so that pedestrians can immediately recognize the approach of the vehicle.

[0157] When a high-risk situation (e.g., RR ≥ threshold, TTC ≤ 2 seconds) is determined, the vehicle integrated terminal (140) can operate automatic emergency braking (AEB) in conjunction with the ECU and brake control device and control the vehicle speed to decelerate or stop.

[0158] Through the series of steps described above, the multimodal sensor fusion-based pedestrian detection and collision risk prediction method according to the present invention can realize more accurate and reliable collision risk prediction and warning by fusing various sensor information and reflecting the pedestrian's behavioral intentions, compared to conventional technology that relies on a single sensor or simple distance-based TTC calculation. Explanation of the symbols

[0159] 100: System 110: Sensor module 120: Multimodal Fusion Section 130: AI-based recognition and prediction unit 140: Vehicle Integrated Terminal 150: Alarm display unit S1: Object recognition step S2: Object tracking step S3: Intention prediction stage S4: Collision Risk Prediction Step

Claims

Claim 1 A vehicle-pedestrian collision prediction system for predicting the risk of collision with a pedestrian or two-wheeled vehicle around a vehicle, comprising: a sensor module unit that collects environmental data from a plurality of sensors having different detection principles, positioned at least one of the front, rear, left, and right sides of the vehicle; A multimodal fusion unit that, for sensor data collected from the sensor module, performs a process of calculating an environment-adaptive weight based on at least one of (a) time alignment to a common reference time, (b) spatial alignment to a vehicle reference coordinate system, and (c) illuminance, weather conditions, frame quality, visibility, sensor reliability, and time-series consistency, and adjusts the weights to increase or decrease the weight of a specific sensor according to environmental conditions, including increasing the weight of the IR camera in a night or low-light environment, increasing the weight of the ultrasonic sensor when rain or snow is detected, and decreasing the weight of the RGB camera when backlight or glare is detected, and then generates an integrated feature map containing spatial information and temporal change information simultaneously by combining feature values ​​of different sensors in the same coordinate system by weighting them at the pixel unit or spatial grid unit using the adjusted weights; An AI-based recognition and prediction unit that calculates a crossing intention probability (P_cross) using a neural network-based time series prediction model that takes the above-mentioned integrated feature map as input and takes as input (i) object recognition, (ii) time series-based object tracking, and (iii) time series features including a change in the pedestrian's head direction (Δθ), body rotation angle, change in speed or acceleration, relative angle with respect to the vehicle approach direction, and spatial distribution of surrounding pedestrians; and a risk index calculation unit that calculates a risk index (RR) defined as RR = P_cross / TTC using the crossing intention probability (P_cross) calculated by the AI-based recognition and prediction unit and the expected time of collision (TTC) calculated based on the relative distance and relative speed between the vehicle and the object.A vehicle-pedestrian collision prediction system characterized by including a vehicle integrated terminal that determines a risk level based on the above risk index (RR) and performs warning output to a driver or pedestrian or vehicle control. Claim 2 delete Claim 3 delete Claim 4 A vehicle-pedestrian collision prediction system according to claim 1, wherein the AI-based recognition and prediction unit is configured with a parallel streaming pipeline structure based on an FPGA or ASIC, and includes an on-device AI computing unit configured to process CNN-based object recognition, BiLSTM or GRU-based time series analysis, Time-To-Collision (TTC) calculation, and Risk Index (RR) calculation within a preset delay time or less. Claim 5 delete Claim 6 A vehicle-pedestrian collision prediction method for predicting the risk of collision with a pedestrian or two-wheeled vehicle around a vehicle, comprising: (a) collecting environmental data using a plurality of sensors having different detection principles positioned at least one of the front, rear, left, and right sides of the vehicle; (b) calculating an environment-adaptive weight based on at least one of illuminance, weather conditions, frame quality, visibility, sensor reliability, and time-series consistency for the sensor data, including time alignment to a common reference time, spatial alignment to a vehicle reference coordinate system; (c) increasing or decreasing the weight of a specific sensor according to environmental conditions, including increasing the weight of an IR camera in a night or low-light environment, increasing the weight of an ultrasonic sensor when rain or snow is detected, and decreasing the weight of an RGB camera when backlight or glare is detected, and generating an integrated feature map by combining the feature values ​​of different sensors in the same coordinate system by weighted summing at the pixel unit or spatial grid unit using the said weights; (d) using the integrated feature map as input for object recognition, time-series-based object tracking, and A method for predicting a vehicle-pedestrian collision, characterized by comprising: a step of calculating a probability of crossing intent (P_cross) using time series features including a change in the pedestrian's head direction (Δθ), a body rotation angle, a change in speed or acceleration, a relative angle with respect to the vehicle approach direction, and a spatial distribution of surrounding pedestrians; (e) a step of calculating a risk index (RR) defined as RR = P_cross / TTC using the probability of crossing intent (P_cross) and a predicted time of collision (TTC) calculated based on the relative distance and relative speed between the vehicle and the object; and (f) a step of determining a risk level based on the risk index (RR) and performing an alarm output or vehicle control. Claim 7 delete

Images