System and method for recognizing traffic objects and controlling traffic signals using radar data and image data based on artificial intelligence

Abstract

The embodiments present a system and method for recognizing traffic objects and controlling traffic signals using radar data and image data based on artificial intelligence. The system may include a traffic information collection device that collects information related to traffic conditions using a radar module and a camera module, a control server that recognizes traffic objects and generates traffic signal control information based on the information collected from the traffic information collection device, and a signal control device that controls a road guidance device based on the traffic signal control information. The above traffic information collection device acquires radar detection data and image data for a preset road area from the radar module and the camera module, transmits sensing information including the radar detection data and the image data to the control server, and the control server determines values ​​for a plurality of detection items based on the radar detection data, calculates a radar reliability index based on the values ​​for the plurality of detection items, activates a radar-centered analysis model as a basic operating mode to determine information about a first traffic object based on the sensing information, maintains the activation of the radar-centered analysis model when the radar reliability index is maintained above a preset first threshold value for a preset first number of frames or more, and activates an image-centered analysis model when the radar reliability index is maintained below a preset second threshold value for a preset second number of frames or more, thereby determining information about a second traffic object through the image-centered analysis model based on the sensing information, the preset first threshold value is a value greater than the preset second threshold value, determines the traffic signal control information based on the information about the first traffic object or the information about the second traffic object, and the traffic signal control Information can be transmitted to the above signal control device.

Description

Technology Field

[0001] The embodiments of the present disclosure relate to a technology for recognizing traffic objects and controlling traffic signals, and to a system and method for recognizing traffic objects and controlling traffic signals using radar data and image data based on artificial intelligence. Background Technology

[0002] With the recent proliferation of autonomous driving and Intelligent Transportation Systems (ITS), the importance of technologies capable of accurately recognizing traffic objects and controlling traffic signals in real time at urban intersections and road sections is rapidly increasing. Conventional traffic control systems typically relied on vehicle detectors, loop sensors, and video cameras to adjust traffic light cycles or monitor traffic flow. However, because these methods rely on a single sensor, they suffer from limitations where detection accuracy deteriorates rapidly depending on weather conditions, changes in illumination, and obstacles.

[0003] For example, video-based traffic surveillance systems are significantly affected by environmental factors such as camera field of view, night lighting, and weather conditions, and face the problem of unstable object boundary recognition during backlighting or rain. On the other hand, while radar-based detection systems can reliably obtain information on distance, speed, and azimuth, it is difficult to distinguish the type or shape of objects, making it difficult to accurately classify various objects such as vehicles, pedestrians, and bicycles.

[0004] For this reason, multimodal recognition technology, which fuses radar and image data to compensate for each other's shortcomings, has recently been attracting attention. However, conventional fusion methods have been limited to simply merging or averaging the outputs of the two sensors, failing to consider sensor reliability in specific situations. For instance, while radar reliability is high in conditions of heavy rain or fog, image quality degrades; however, existing fusion systems fail to reflect this in real-time and only perform weighted sums at a fixed ratio, which can lead to misrecognition or signal control errors.

[0005] In addition, existing traffic object recognition systems control traffic volume based on fixed signal cycles and predefined patterns, making it difficult to respond immediately to dynamic situations such as congestion in a specific direction or the sudden appearance of pedestrians. In particular, even in some systems where AI models are applied, there was a problem where recognition accuracy became unstable because a single model processed all situations, maintaining the same inference path even when sensor reliability deteriorated.

[0006] Accordingly, there is a need for a system and method that recognizes traffic objects and controls traffic signals using radar data and image data based on artificial intelligence. The problem to be solved

[0007] Embodiments of the present disclosure can provide a system and method for recognizing traffic objects and controlling traffic signals using radar data and image data based on artificial intelligence.

[0008] The technical problems to be solved in the embodiments are not limited to those mentioned above, and other unmentioned technical problems may be considered by those skilled in the art from the various embodiments described below. means of solving the problem

[0009] A system for recognizing traffic objects and controlling traffic signals using radar data and image data based on artificial intelligence according to one embodiment may include a traffic information collection device that collects information related to traffic conditions using a radar module and a camera module, a control server that recognizes traffic objects and generates traffic signal control information based on the information collected from the traffic information collection device, and a signal control device that controls a road guidance device based on the traffic signal control information. The above traffic information collection device acquires radar detection data and image data for a preset road area from the radar module and the camera module, transmits sensing information including the radar detection data and the image data to the control server, and the control server determines values ​​for a plurality of detection items based on the radar detection data, calculates a radar reliability index based on the values ​​for the plurality of detection items, activates a radar-centered analysis model as a basic operating mode to determine information about a first traffic object based on the sensing information, maintains the activation of the radar-centered analysis model when the radar reliability index is maintained above a preset first threshold value for a preset first number of frames or more, and activates an image-centered analysis model when the radar reliability index is maintained below a preset second threshold value for a preset second number of frames or more, thereby determining information about a second traffic object through the image-centered analysis model based on the sensing information, the preset first threshold value is a value greater than the preset second threshold value, determines the traffic signal control information based on the information about the first traffic object or the information about the second traffic object, and the traffic signal control Information can be transmitted to the above signal control device. Effects of the invention

[0010] According to the embodiments, the present disclosure enables precise recognition of traffic objects on a road and automatic control of traffic signals based on this by fusion analysis of data collected from a radar module and a camera module through an artificial intelligence model. Accordingly, problems such as detection instability, dependence on lighting, and weather influences that occurred in conventional image-only recognition systems or radar-only detection systems can be minimized.

[0011] According to the embodiments, the present disclosure can automatically switch to an optimal recognition mode depending on environmental conditions by selectively operating a radar-centric analysis model and an image-centric analysis model. Accordingly, traffic object recognition performance can be stably maintained even in abnormal scenarios such as bad weather, nighttime, and backlighting environments, and unnecessary computational load can be reduced and system resources efficiently managed through a reliability-based mode switching structure.

[0012] According to the embodiments, the present disclosure calculates the location, speed, size, direction of travel, and object class of traffic objects in real time from the results of fused artificial intelligence analysis, and by having a control server use this information to automatically determine traffic signal control information, various traffic events such as vehicle queues, pedestrian occupancy rates, exceeding section speed limits, and garage restrictions can be preemptively controlled. Accordingly, it provides the effect of alleviating intersection congestion, optimizing traffic flow, and reducing the possibility of accidents.

[0013] The effects obtainable from the embodiments are not limited to those mentioned above, and other unmentioned effects can be clearly derived and understood by a person skilled in the art based on the detailed description below. Brief explanation of the drawing

[0014] The accompanying drawings, included as part of the detailed description to aid in understanding the embodiments, provide various embodiments and explain the technical features of the various embodiments together with the detailed description. FIG. 1 is a schematic diagram of a system that recognizes traffic objects and controls traffic signals using radar data and image data based on artificial intelligence according to one embodiment. FIG. 2 is a flowchart illustrating the operation of a control server in a system that recognizes traffic objects and controls traffic signals using radar data and image data based on artificial intelligence according to one embodiment. FIG. 3 is a flowchart illustrating the operation of activating an artificial intelligence model in a system that recognizes traffic objects and controls traffic signals using radar data and image data based on artificial intelligence according to one embodiment. FIG. 4 is a block diagram showing the configuration of a control server according to one embodiment. Specific details for implementing the invention

[0015] The following embodiments are combinations of the components and features of the embodiments in a specific form. Each component or feature may be considered optional unless otherwise explicitly stated. Each component or feature may be implemented in a form not combined with other components or features. Additionally, various embodiments may be constructed by combining some components and / or features. The order of operations described in various embodiments may be changed. Some components or features of one embodiment may be included in another embodiment, or may be replaced with corresponding components or features of another embodiment.

[0016] In the description of the drawings, procedures or steps that could obscure the essence of the various embodiments are not described, nor are procedures or steps that can be understood by a person of ordinary knowledge in the relevant technical field described.

[0017] Throughout the specification, when a part is described as "comprising" or "including" a component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "...part," "...unit," and "module" as used in the specification refer to a unit that performs at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software. Additionally, "one (a or an)," "one," "the," and similar related terms may be used in the context describing various embodiments (particularly in the context of the following claims) in both singular and plural forms, unless otherwise indicated in the specification or clearly contradicted by the context.

[0018] Hereinafter, embodiments according to various examples will be described in detail with reference to the accompanying drawings. The detailed description disclosed below, together with the accompanying drawings, is intended to describe exemplary embodiments of various examples and is not intended to represent the only embodiment.

[0019] In addition, specific terms used in various embodiments are provided to aid in understanding the various embodiments, and the use of such specific terms may be modified in other forms within the scope of not departing from the technical concept of the various embodiments.

[0020] FIG. 1 is a schematic diagram of a system for recognizing traffic objects and controlling traffic signals using radar data and image data based on artificial intelligence according to one embodiment. The embodiment of FIG. 1 may be combined with various embodiments of the present disclosure.

[0021] Referring to FIG. 1, a system (100) (hereinafter, system (100)) that recognizes traffic objects and controls traffic signals using radar data and image data based on artificial intelligence may include a traffic information collection device (110), a control server (120), and a signal control device (130).

[0022] The system (100) may be a system that collects traffic conditions on the road in real time through a traffic information collection device (110) installed on the road, and a control server (120) recognizes traffic objects such as vehicles, pedestrians, and motorcycles based on the collected information, and automatically controls a road guidance device through a signal control device (130) based on the location and speed information of the recognized traffic objects.

[0023] For example, the control server (120) can control the extension of the green signal time for a specific direction when traffic congestion in that direction increases. For example, the control server (120) can control the extension of the red signal time for vehicle signals when a pedestrian has entered the crosswalk. Additionally, when an emergency situation, such as an accident, is detected, the control server (120) can send an emergency notification message to the user terminal to enable a rapid response. At this time, the user terminal is a terminal registered as the person in charge of the road area and can check traffic conditions, signal change requests, or anomaly detection information provided by the control server (120) in real time and send manual control commands. For example, the user terminal can be implemented as various devices including a smartphone, mobile phone, smart TV, set-top box, tablet PC, laptop computer, desktop, wearable device, etc.

[0024] The traffic information collection device (110) may be a device that collects information related to traffic conditions using a radar module and a camera module. For example, the traffic information collection device (110) may be installed on a road structure and may be mechanically aligned so that the transmission and reception of radar signals and the capture of images occur simultaneously at the same location.

[0025] For example, traffic information collection devices (110) may be distributed and installed at multiple points on an intersection or a multi-lane road section. For example, based on a single intersection, traffic information collection devices (110) may be placed in the left lane area, the center straight lane area, and the right lane area, respectively, and each traffic information collection device (110) may be installed to cover a preset azimuth range.

[0026] For example, the traffic information collection device (110) can acquire radar detection data and image data for a preset road area from a radar module and a camera module. The traffic information collection device (110) can transmit sensing information including radar detection data and image data to a control server (120).

[0027] Additionally, for example, the traffic information collection device (110) can transmit the sensing information, time information, and device identification information together to the control server (120).

[0028] The radar module can receive signals reflected from a target object using the Frequency Modulated Continuous Wave (FMCW) method. At this time, the radar module can generate radar detection data including distance (r), relative velocity (vr), azimuth (θ), and reflection intensity (Radar Cross Section) values ​​for the target object based on the received signals. Here, distance is a value calculated based on the time it takes for the electromagnetic wave to travel back and forth between the radar and the target object, and may represent the straight-line distance from the radar module's reference point to the target object. For example, distance may be expressed in meters. Relative velocity is a value calculated based on the change in Doppler frequency included in the signal reflected from the target object, and may represent the relative speed of movement of the target object as it approaches or moves away from the radar module. For example, relative velocity may be expressed as a positive or negative number in meters per second. In this case, a positive number may indicate a relative speed of movement away, and a negative number may indicate a relative speed of movement approaching. The azimuth is a value representing the horizontal direction in which a target object is located within the radar reference coordinate system, and can be calculated based on the phase difference or arrival time difference between reflected signals received by each receiving antenna. For example, the azimuth can be expressed in degrees (°). The reflection intensity value represents the strength of radar energy reflected back by the target object, and may vary depending on the object's size, shape, material, and angle of incidence. For example, the reflection intensity value can be expressed in decibel square meters (dBsm).

[0029] For example, a radar module may include a transmitter comprising one or more transmitting antennas and a receiver comprising a plurality of receiving antennas. Through this antenna array, the radar module can calculate distance and relative speed, as well as azimuth information based on the phase difference between the receiving antennas. For example, the transmitter of the radar module may radiate a signal with linear frequency modulation applied to it into the corresponding road area, and the receiver of the radar module may calculate distance and speed by analyzing the difference in reflected frequencies and the Doppler frequency. For example, a single radar point included in the radar detection data may be set to (r, θ, vr, RCS, t). The radar module may generate radar detection data in frames according to a preset period (e.g., 20 ms or 50 ms).

[0030] The camera module can acquire image data by capturing the same road area as the radar module in the traffic information collection device (110). For example, the camera module may include a lens, an image sensor (CMOS or CCD), and an ISP (Image Signal Processor). For example, the exposure time, white balance, and focus value for the camera module may be synchronized with the operation cycle of the radar module. For example, the camera module may capture the road area on a frame-by-frame basis and transmit the captured image data to the control server (120) by including it in the sensing information.

[0031] For example, the frame rates of the radar module and the camera module can be synchronized to the same frame rate. In this case, the frame rate can be set between 15 fps (frames per second) and 30 fps.

[0032] Additionally, for example, the camera module can automatically adjust the frame acquisition rate (FPS) according to the ambient lighting environment, and IR lighting or HDR shooting methods can be applied in nighttime environments.

[0033] For example, the traffic information collection device (110) may further include a control unit including a processor, memory, and a communication module.

[0034] The processor can execute software to control at least one other component (e.g., a hardware or software component) of a device connected to the processor and perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor can store instructions or data received from other components (e.g., a battery or a communication module) in volatile memory, process the instructions or data stored in volatile memory, and store the resulting data in non-volatile memory. According to one embodiment, the processor may include a main processor (e.g., a central processing unit or an application processor) or an auxiliary processor (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor.

[0035] For example, if a main processor and an auxiliary processor are included, the auxiliary processor may be configured to use less power than the main processor or to be specialized for a designated function. The auxiliary processor may be implemented separately from the main processor or as part thereof. The auxiliary processor may control at least some of the functions or states associated with at least one of the components of the system (100), for example, on behalf of the main processor while the main processor is in an inactive (e.g., sleep) state, or together with the main processor while the main processor is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor may be implemented as part of another functionally related component. According to one embodiment, the auxiliary processor may include a hardware structure specialized for processing an artificial intelligence model.

[0036] For example, the processor may use an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the device itself where the artificial intelligence model is executed (e.g., traffic information collection device (110)), or through a separate server (e.g., control server (120)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers. An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of one or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0037] The memory can store various data used by at least one component of the traffic information collection device (110). The data may include, for example, input data or output data for software and related commands. The memory may include volatile memory or non-volatile memory. A program may be stored in the memory as software and may include, for example, an operating system, middleware, or an application.

[0038] The communication module may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an external device or a control server (120) and a traffic information collection device (110), and the performance of communication through the established communication channel. The communication module may include one or more communication processors that operate independently of a processor (e.g., an application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module) or a wired communication module (e.g., a LAN (local area network) communication module, or a power line communication module). Among these communication modules, the corresponding communication modules can communicate with an external device through a first network (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a long-range communication network such as a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips).

[0039] The wireless communication module can support 5G networks following 4G networks and next-generation communication technologies, such as new radio access technology (NR access technology). NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module can support high-frequency bands (e.g., mmWave band) to achieve high data transmission rates, for example. The wireless communication module can support various technologies to secure performance in high-frequency bands, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beamforming, or large-scale antenna. The wireless communication module can support various requirements specified in external electronic devices or network systems. According to one embodiment, the wireless communication module can support a Peak data rate for eMBB realization (e.g., 20 Gbps or higher), loss coverage for mMTC realization (e.g., 164 dB or lower), or U-plane latency for URLLC realization (e.g., downlink (DL) and uplink (UL) each 0.5 ms or lower, or round trip 1 ms or lower).

[0040] At least some of the components of the traffic information collection device (110) may be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and may exchange signals (e.g., commands or data) with each other.

[0041] The first network and the second network refer to a connection structure capable of exchanging information between each node, such as terminals and servers, or a network connecting a traffic collection device (110), a control server (120), a user terminal, and a system (100). The first network and the second network include, but are not limited to, the Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), 3G, 4G, LTE, 5G, Wi-Fi, etc. The first network and the second network may be closed first network and second networks such as LAN or WAN, but it is preferable that they be open, such as the Internet. The Internet refers to a global open computer network structure comprising a first network and a second network that provides protocols such as TCP / IP, TCP, and UDP (User Datagram Protocol), as well as various services existing at the upper layers, namely HTTP (HyperText Transfer Protocol), Telnet, FTP (File Transfer Protocol), DNS (Domain Name System), SMTP (Simple Mail Transfer Protocol), SNMP (Simple Network Management Protocol), NFS (Network File Service), and NIS (Network Information Service).

[0042] A database may have a general data structure implemented in the storage space (hard disk or memory) of a computer system using a database management program (DBMS). A database may have a data storage form that allows for the free retrieval (extraction), deletion, editing, and addition of data. A database may be implemented to suit the purpose of an embodiment of the present disclosure using a relational database management system (RDBMS) such as Oracle, Informix, Sybase, and DB2, an object-oriented database management system (OODBMS) such as Gemston, Orion, and O2, and an XML native database such as Excelon, Tamino, and Sekaiju, and may have appropriate fields or elements to achieve its functions.

[0043] The control server (120) may be a server that recognizes traffic objects and controls traffic signals based on information collected from the traffic information collection device (110).

[0044] For example, the control server (120) can receive sensing information from the traffic information collection device (110) and determine the location, speed, size, direction of travel, and object class of a traffic object by fusion analysis of radar detection data and image data through an artificial intelligence model. For example, the control server (120) can activate one of a plurality of artificial intelligence models (e.g., a radar-centric analysis model or an image-centric analysis model) based on the reliability index of the radar detection data. For example, the control server (120) can calculate information about the traffic object using the activated artificial intelligence model and control the signal cycle of an intersection or the operating state of a road guidance device according to the calculated information about the traffic object. For example, the control server (120) can determine the location and speed of the traffic object by deriving radar-based object candidates using radar detection data and performing image-based correction or tracking on the object candidates.

[0045] For example, the control server (120) can generate traffic signal control information for use by the signal control device (130) based on information about the traffic object.

[0046] Additionally, for example, the control server (120) can determine whether an emergency situation has occurred based on information about the traffic object. For example, if an emergency situation occurs, the control server (120) can send a notification message to a pre-configured user terminal.

[0047] Additionally, for example, the control server (120) can store and update status information for each traffic object in frames, including an identification number (ID) for each traffic object, movement trajectory, whether it is stopped, whether it is approaching pedestrians, and whether it is out of lane.

[0048] For example, the control server (120) can determine traffic volume, the number of waiting vehicles, whether pedestrians are entering, and whether emergency vehicles are approaching based on status information regarding traffic objects. Accordingly, the control server (120) can determine a signal pattern that extends the green signal time or the red signal time of a traffic light. For example, the control server (120) can increase the left-turn signal time if the number of vehicles waiting in the left-turn lane exceeds a preset threshold. For example, the control server (120) can extend the crosswalk signal if the number of pedestrians waiting on the sidewalk exceeds a preset threshold. For example, the control server (120) can grant priority signals to the lane in the direction of travel when an emergency vehicle approaches. Additionally, for example, the control server (120) can analyze the movement speed and trajectory of an object and, if a road emergency situation (e.g., sudden braking, driving in the wrong direction, sudden entry of a pedestrian, or risk of collision) is detected, generate an alarm message and transmit it to at least one of the signal control device (130) or user terminal.

[0049] For example, the control server (120) may be an intelligent server using machine learning and / or neural networks. In terms of hardware, the control server (120) may have the same configuration as a conventional web server or service server. However, in terms of software, it may include program modules that perform various functions and are implemented through various languages ​​such as C, C++, Java, Python, Golang, and Kotlin. Furthermore, the control server (120) generally refers to a computer system that is connected to an unspecified number of clients and / or other servers through an open computer network such as the Internet, receives requests for task execution from clients or other servers, and derives and provides the results of the tasks, as well as computer software (server programs) installed for this purpose. In addition, the control server (120) should be understood as a broad concept that includes, in addition to the aforementioned server programs, a series of application programs running on the server and, in some cases, various databases (DB: Database, hereinafter referred to as "DB") built internally or externally. Accordingly, the control server (120) classifies data, stores it in a DB, and manages it, and this DB can be implemented inside or outside the server. Additionally, the control server (120) can be implemented on general server hardware using various server programs provided according to operating systems such as Windows, Linux, UNIX, and Macintosh, and representative examples include IIS (Internet Information Server) used in a Windows environment and CERN, NCSA, APPACH, TOMCAT used in a UNIX environment to implement web services.In addition, the control server (120) may be linked with a user authentication system for the service or an authentication system related to the service.

[0050] The signal control device (130) may be a device that controls the road guidance device based on traffic signal control information received from the control server (120).

[0051] For example, the signal control device (130) can control the signal cycle per intersection, the left turn signal duration, and the pedestrian signal display time, and can transmit the output signal to the road guidance device.

[0052] For example, the signal control device (130) can apply an electrical signal to a road guidance device including a traffic light, a road electronic display, a lane guidance indicator, or a pedestrian signal device installed at an intersection, and change at least one of the lighting color, flashing cycle, or display text according to at least one of the signal cycle per intersection, the left turn signal duration, or the pedestrian signal display time.

[0053] For example, if the signal control device (130) receives a green signal extension command from the control server (120), it can increase the green display time of the traffic light in that direction. For example, if the signal control device (130) receives a pedestrian approach warning command from the control server (120), it can display the phrase “Pedestrian Warning” on the road electronic display board.

[0054] Additionally, for example, the signal control device (130) may monitor the signal light status (red, green, yellow), the signal lighting time, and the voltage / current status of the signal control device (130) in real time, and may transmit fault information to the control server (120) if an abnormal condition is detected.

[0055] The control server (120) and the signal control device (130) can transmit and receive data via wired Ethernet communication, LTE / 5G communication, or a dedicated optical communication network, and data integrity verification and encryption processing (TLS (Transport Layer Security) and VPN (Virtual Private Network) methods) can be performed during the communication process. For example, the control server (120) can ensure data integrity and confidentiality by applying a TLS-based encryption protocol. For example, by establishing a VPN connection, the control server (120) can perform data protection at the internal network level by forming a secure communication path identical to that of a private network even on an internet-based public network.

[0056] Additionally, for example, the control server (120) can check the response time and control reflection of each signal control device (130), and in the event of non-response or failure, perform the function of retransmitting the same control command through a backup control path or switching to an emergency flashing mode.

[0057] For example, the signal control device (130) may include a local control program that automatically returns to a basic signal pattern in case communication with the control server (120) is interrupted.

[0058] FIG. 2 is a flowchart illustrating the operation of a control server in a system that recognizes traffic objects and controls traffic signals using radar data and image data based on artificial intelligence according to one embodiment. The embodiment of FIG. 2 can be combined with various embodiments of the present disclosure.

[0059] Referring to FIG. 2, in step S210, the control server can determine values ​​for multiple detection items based on radar detection data.

[0060] Radar detection data is a radar point cloud acquired by a radar module using the FMCW method, and each radar point may include distance, relative velocity, azimuth, and reflection intensity values. For example, if radar detection data is acquired for 10 frames, the radar detection data may include an independent radar point cloud for each frame. That is, the radar detection data can be represented as 10 frame-unit data consisting of {P1, P2, … , P10}, and each Pt is {(r i , v ri , θ i , RCS i ) | i = 1… N t It can be composed of a set of N points of the form}. Here, N t t can be the total number of radar points detected in frame t. Radar detection data configured in this way can be continuously updated according to a preset period (e.g., 20ms, 50ms, etc.).

[0061] Here, a radar point may be a minimum unit detection result representing the intensity peak of a reflected signal detected at a specific distance, speed, or azimuth angle by the combination of the radar module's transmitting and receiving antennas and frequency modulation signals. In this case, since a single actual object (e.g., a vehicle or a pedestrian) can generate different reflected signals from multiple reflective surfaces, multiple radar points corresponding to that object may be formed within a single frame. Therefore, the target object may be the result of clustering (e.g., DBSCAN, Mean-Shift) multiple radar points that are spatially adjacent within the same frame and have similar speed and reflected intensity characteristics, and integrating them into a single radar point. For example, if the front bumper, roof, and side reflected signals of the same vehicle are each detected as independent points, they can be integrated into a single radar point based on proximity in distance, azimuth angle, and speed, as well as similarity in reflected intensity. For example, a control server can calculate the difference in distance coordinates, azimuth angle, relative speed, and reflected intensity between radar points, and group points where each difference value is below a preset threshold into the same cluster. Clustering can be performed to minimize multidimensional Euclidean distances based on distance and velocity axes. For example, preset thresholds for clustering can be set such that the distance threshold is 0.5 m or less, the azimuth threshold is 1 degree or less, the velocity threshold is 1.0 m / s or less, and the reflection intensity threshold is 3 dB or less. Multiple radar points satisfying these conditions are considered as the same reflector (object), and the average distance, average azimuth, average velocity, and average reflection intensity values ​​can be integrated as attributes of a representative radar point.

[0062] For example, whether multiple temporally consecutive frames are the same target object can be determined based on whether the distance coordinates, velocity vectors, and reflection intensity values ​​of radar points determined to be the same identification candidate each fall within a preset error tolerance range. For example, if the difference in distance, velocity, and reflection intensity between a first radar point and a second radar point located near the same position in consecutive first and second frames is less than or equal to the distance tolerance, velocity tolerance, and reflection tolerance, respectively, the two radar points can be determined to be the same object.

[0063] For example, the control server can determine the values ​​for multiple detection items for each frame based on radar detection data.

[0064] Values ​​for multiple detection items may include values ​​for average reflection intensity, Doppler consistency, distance-velocity physical consistency, and object count stability.

[0065] The value for average reflection intensity can be set based on the average and standard deviation values ​​calculated by aggregating the reflection intensity values ​​of all radar points detected within a frame. By comparing the value for average reflection intensity with a preset normal range, it can be used as a reference value to determine whether there is a decrease in radar module sensitivity or an anomaly in the external environment, such as rainfall, snow, or fog. The reflection intensity value can be expressed in decibel square meters (dBsm). For example, the average value (RCS avg )silver It can be calculated as follows. Standard deviation value (σ RCS )silver It can be calculated as follows. Here, N may be the total number of radar points included in the radar point cloud corresponding to a single frame. The value for the average reflection intensity is It can be determined as. In this case, σ maxθ can be the maximum standard deviation reference value in a steady state. That is, the value for average reflection intensity can be normalized to a range of 0 to 1, and the closer it is to 1, the more stable the radar reflection intensity distribution may be.

[0066] The value for Doppler coherence is the amount of change in relative velocity (Δv) measured for the same object between consecutive frames. r It can be calculated based on ). Δv r If this preset threshold range is exceeded, the frame may be determined to have low Doppler consistency. For example, the control server determines the relative velocity values ​​of radar points identified as the same object in consecutive first and second frames as v, respectively. r (t) , v r (t-1) Set to, and the change in relative velocity Δv r = It can be calculated. In this case, if an object's relative velocity change is less than or equal to a preset threshold (e.g., 5 m / s or 3σ statistical range), it may be determined to be an object with stable Doppler characteristics. For example, the value for Doppler consistency can be determined by dividing the number of stable objects in the frame by the total number of objects in the frame plus 1. The value for Doppler consistency ranges from 0 to 1, and a value closer to 1 may indicate that the velocity change between consecutive frames is physically consistent.

[0067] The value for distance-velocity physical consistency can be calculated by determining whether the physical relationship between distance information and velocity information is violated, such as when an abnormal velocity is detected for an object classified as stationary, or when the velocity value is calculated as negative despite the object being moving toward the radar. For example, the control server may determine an object as having a physical discrepancy if the velocity value of an object classified as stationary in the relevant frame exceeds 0 m / s, or if an object located at a close distance (e.g., 0–5 m) is detected as having a negative velocity value despite approaching the radar. For instance, the distance-velocity physical consistency value can be calculated as the ratio of the number of objects not determined to have a physical discrepancy within the frame to the total number of objects within the frame. The value for distance-velocity physical consistency ranges from 0 to 1, and a value closer to 1 may indicate a physically valid state for the distance-velocity relationship.

[0068] The value for object count stability can be calculated based on the change in the total number of radar points included in the radar detection data. If the number of objects decreases or increases sharply between frames, it is determined to be an anomaly detection caused by sensor shielding, ambient interference, or false reflections, and the object count stability value may be calculated as low. For example, the object count stability value (Scorecount) is It can be calculated as follows. Here, N t is the number of objects in frame t, and Nt-1 may be the number of objects in frame t-1. In this case, the min function may be a function that selects the smaller of the two values ​​inside the parentheses. The value for object count stability ranges from 0 to 1, and the closer it is to 1, the more stable the fluctuation in the number of detected objects between frames may be.

[0069] For example, a traffic information collection device can acquire radar detection data and image data for a preset road area from a radar module and the camera module. The preset road area is a detection section divided into intersections or lanes, and can be set based on the transmission signal bandwidth of the radar module, the structure of the transmitting antenna and receiving antenna array, and the field of view of the camera module. For example, if the radar module uses the FMCW method, the detectable distance range and azimuth range can be determined according to the chirp sweep bandwidth of the transmission frequency and the azimuth resolution of the antenna array. For example, if the transmission frequency center is 77 GHz and the modulation bandwidth is 1 GHz, the distance resolution can be set to approximately 0.15 m, and if a MIMO array with a 12 x 16 channel configuration is used, the azimuth resolution can be set to approximately 1 to 2 degrees. For example, a single road area can be set as a fan-shaped area with a range of ±30 degrees to the left and right relative to 0 degrees forward and a distance of 0 to 60 m, and multiple traffic information collection devices within the same intersection can be configured to cover adjacent areas complementarily.

[0070] For example, traffic information collection devices may be deployed in the left lane area, the center straight lane area, and the right lane area, respectively, based on a single intersection, and each device may be installed to cover a preset azimuth range. For instance, multiple traffic information collection devices within the same intersection can minimize the omission of traffic object detection across the entire road area by covering adjacent detection areas complementarily. For instance, the first traffic information collection device may detect vehicles in the left-turn lane direction (-30° to 0°), the second traffic information collection device may detect vehicles in the straight lane direction (0° to +30°), and the third traffic information collection device may be responsible for the right-turn lane direction (+30° to +60°). The control server can analyze the traffic flow of the entire intersection by integrating radar detection data and image data received from the multiple traffic information collection devices into time-synchronized sensing information.

[0071] For example, a traffic information collection device can transmit sensing information, including radar detection data and image data, to a control server.

[0072] Sensing information can be transmitted along with metadata containing time information and device identification information.

[0073] Time information is a time stamp indicating the point in time when the traffic information collection device acquired sensing information, and can be expressed, for example, as a UTC (Coordinated Universal Time) standard time code in the format YYYY-MM-DD hh:mm:ss.sss. Time information can be used as a standard to ensure that the operation of the radar module and the camera module is synchronized on a frame-by-frame basis. Radar detection frames and image frames having the same time stamp can be set as data acquired at the same point in time.

[0074] Device ID is information for uniquely distinguishing a traffic information collection device and can be generated as a string combining a device manufacturer code, an installation location code, and a device serial number. For example, device ID can be expressed in the form of RDR-ICX01-0003, where RDR indicates that it is a radar-based traffic information collection device, ICX01 indicates an intersection location code, and 0003 indicates an individual device serial number.

[0075] The traffic information collection device can transmit sensing information with added metadata through a TLS (Transport Layer Security)-based encrypted communication channel.

[0076] For example, data transmission and reception between a control server and a traffic information collection device can be performed through a TLS session including a public key-based authentication procedure, and when traversing an external network, a VPN (Virtual Private Network) tunnel is established so that all traffic can be transmitted through an encrypted virtual private network path.

[0077] Therefore, the traffic information collection device can support the control server in performing time-based alignment analysis and reliability calculation by transmitting time-synchronized radar data and image data, along with metadata for identifying such data, to the control server through a secure communication channel.

[0078] In step S220, the control server can calculate a radar reliability index based on the values ​​for multiple detection items.

[0079] The Radar Reliability Index may be a quantitative indicator that comprehensively reflects the sensitivity, frequency stability, detection consistency, and output stability of a radar module.

[0080] For example, the control server can calculate the radar reliability index by weighting and summing the values ​​for average reflection intensity, Doppler consistency, distance-velocity physical consistency, and object count stability, respectively.

[0081] The weights (w1, w2, w3, w4) for each item can be set to reflect the relative influence on radar detection reliability. For example, each weight can be determined by comprehensively considering the sensitivity of each detection item, responsiveness to environmental changes, and the contribution rate of measurement error, and each weight can be set to a value between 0 and 1, and the sum of all weights can be normalized to 1.

[0082] For example, the average reflection intensity item is the most sensitive to changes in external conditions such as rainfall, snow cover, and fog; therefore, its weight w1 may be set relatively high to primarily reflect the impact of radar sensitivity degradation and environmental noise on detection reliability. Since the Doppler consistency item is directly related to the stability of velocity changes between consecutive frames, its weight w2 may be set to an intermediate level when emphasizing the reliability of velocity measurements. The distance-velocity physical consistency item is used to evaluate the sensor's operating state and the fit of the physical model; as it reflects the proportion of measured distance and velocity data that maintain a physically valid relationship, its weight w3 may be set to a level equal to or similar to that of Doppler consistency. The object count stability item is used to evaluate the rate of variation in the number of detected objects between frames; since natural fluctuations may occur due to external factors such as vehicle entry or exit from the detection area, its influence is evaluated as relatively low, and its weight w4 may be set to the smallest value. For example, w1 can be set to 0.3, w2 to 0.25, w3 to 0.25, and w4 to 0.2.

[0083] For example, the control server can calculate a radar reliability index for radar detection data for each frame.

[0084] Additionally, for example, the weights for each item may be dynamically corrected using a calibration process based on training data or field verification data.

[0085] For example, the radar reliability index can be set to a real value between 0 and 1, and a value closer to 1 indicates that the detection state of the radar module is stable, while a value closer to 0 indicates that there is a detection anomaly or environmental interference.

[0086] In step S230, the control server can activate a radar-centered analysis model as a basic operating mode to determine information about the first traffic object based on sensing information.

[0087] Here, the basic operating mode may be a basic operating state in which the control server recognizes traffic objects based on radar detection data received from a traffic information collection device.

[0088] The radar-centric analysis model may be a deep learning-based fusion analysis model for determining object information by detecting object candidates within a radar point cloud based on radar detection data, extracting an image region corresponding to the location of the detected object candidates as a low-resolution ROI (Region of Interest) image, and combining radar point features and ROI image features.

[0089] For example, the radar-centric analysis model can determine at least one first traffic object candidate based on the radar point cloud included in the radar detection data. The radar-centric analysis model can determine information about the first traffic object by extracting an image region corresponding to the location of at least one first traffic object candidate as a low-resolution Region of Interest (ROI) image.

[0090] Information regarding the first traffic object is a set of parameters that physically represent the state of the actual traffic object recognized based on radar detection data, and may include the location, speed, size, direction of travel, and object class of each traffic object.

[0091] In step S240, the control server may maintain activation for the radar center analysis model if the radar reliability index is maintained above a preset first threshold value for a preset first number of frames or more.

[0092] For example, the control server can continue the activation of the radar center analysis model if the radar reliability index is maintained above a preset first threshold value (e.g., 0.8 or higher) for a preset first number of frames (e.g., 10 frames or higher).

[0093] A preset first threshold value can be set as a real value between 0 and 1 as a reference value for determining whether the radar reliability index maintains a stable detection state. At this time, the preset first threshold value can be adjusted according to the on-site traffic environment (e.g., weather, vehicle density, radio wave reflectance).

[0094] The first preset number of frames may refer to a temporal verification interval for verifying whether the radar reliability index is maintained stably for a certain period of time. For example, if the radar module generates detection data at 5 frames per second, the first preset number of frames may be set to 10 to 30 frames. In this case, it can be determined whether the radar reliability index is maintained above a threshold value based on the cumulative detection state over approximately 1 to 3 seconds. As such, the first preset number of frames can be set as a continuous verification unit to determine the sensor's stable state by averaging short-term variability and preventing misjudgment caused by transient reflection anomalies or instantaneous noise in a single frame. At this time, the first preset number can be adjusted according to the on-site traffic environment (e.g., weather conditions).

[0095] In addition, for example, if the radar reliability index is maintained above a preset first threshold value for a preset first number of frames or more, and the image-centered analysis model is activated, the control server may deactivate the image-centered analysis model and activate the radar-centered analysis model.

[0096] According to one embodiment, the radar center analysis model can generate radar point vectors for each of the first set number of frames by converting the distance and azimuth angle for each radar point into planar coordinates based on each radar point cloud collected during a first set number of frames.

[0097] A radar point vector may include a position vector and a radar feature vector for at least one radar point. The position vector is a value representing the spatial position of a radar point in a radar reference coordinate system, and can be calculated using distance (r) and azimuth (θ) in the form of an x-cosθ and a y-cosθ. Such a position vector can be used as basic coordinate information to represent the relative position of a target object on a plane.

[0098] The radar feature vector is a set of physical characteristic values ​​measured by the radar module for a radar point, including velocity (v x , v y It may include ), reflection intensity values, and time stamps. In this case, the velocity is calculated using the relative velocity and azimuth (θ), and the velocity at the x-coordinate is v d *cosθ, the velocity at the y-coordinate is v d It can be calculated in the form of *sinθ.

[0099] The radar-centered analysis model can generate a spatiotemporal feature map reflecting velocity changes and movement patterns between frames through a transformer-based encoding network based on a preset first number of radar point vectors.

[0100] A transformer-based encoding network is a frame sequence {P with a first number of consecutive radar point vectors t-K ,P t-K+1 , … ,P t By taking} as input, temporal changes such as the positional movement, velocity change, and reflection intensity change of radar points between frames can be learned through an attention mechanism. In other words, by expressing how each radar point moves over time as a correlation between vectors, dynamic patterns rather than static information of a single frame can be learned.

[0101] For example, a transformer-based encoding network can convert all radar vectors of each frame into a time-series input token sequence by embedding them into fixed-dimensional vectors through an embedding layer.

[0102] That is, the transformed radar point vectors are arranged in chronological order, time series input token sequence Fin=[f t-K , … ,f t It can be composed of ], and each token may include the location, speed, reflection intensity, and time stamp of the radar point.

[0103] Subsequently, the Transformer-based encoding network can calculate the interrelationships between radar point vectors existing in different frames by performing Multi-Head Self-Attention operations. Each radar point vector calculates a similarity score (Attention Score) with other point vectors highly likely to correspond to the same object using multiple attention heads, and can integrally learn movement trajectories, velocity consistency, and changes in reflection intensity across the entire time axis through a weighted sum proportional to that similarity. In this case, for example, similarity can be calculated by scaling the inner product result of a query vector and a key vector and normalizing it into a probabilistic form by applying a softmax function. That is, by quantifying the similarity between each radar point vector and a point vector at a different time point, information from frames with higher importance can be reflected more significantly based on that similarity. For example, by using an attention mechanism to evaluate whether a radar point at a specific time point t has a high correlation with any radar points at previous times points t?1 and t?2, it is possible to learn time-continuous movement patterns rather than static features of a single frame.

[0104] In addition, Transformer-based encoding networks may incorporate positional encoding to recognize the order of each frame. Positional encoding inserts relative interval information along the time axis in the form of vectors based on sine and cosine functions, enabling the learning of continuous changes according to the frame order.

[0105] The output of each computed attention head is integrated through a feed-forward network after undergoing non-linear transformation and normalization, ultimately generating a spatio-temporal integrated feature vector, or spatio-temporal feature map, that reflects the temporal associations of all frames. At this time, the output of each computed attention head can be integrated through a feed-forward network after undergoing a transformation and normalization process applying a non-linear activation function (ReLU (Rectified Linear Unit) or GELU (Gaussian Error Linear Unit)). The ReLU function is an activation function that imparts non-linearity by blocking negative inputs to zero, while the GELU function is an activation function that performs a smoother non-linear transformation by smoothing input values ​​based on a Gaussian probability distribution. Through this, a high-dimensional feature space can be constructed that reflects not only static radar point characteristics at a single viewpoint but also inter-frame trajectory, velocity continuity, and RCS stability. This spatiotemporal feature map serves as a vector representation that integrally expresses the time-series trajectory, direction of movement, acceleration change, and reflection intensity change of the same object, and can be utilized as an input for subsequent object candidate detection and image fusion operations.

[0106] A radar-centered analysis model can determine, based on a spatiotemporal feature map, location coordinates where the probability of an object's existence is greater than or equal to a preset threshold probability as the location coordinates of at least one first traffic object candidate.

[0107] The probability of an object's existence refers to the probability value indicating the presence of an actual object at each location vector on a spatiotemporal feature map, and can be calculated by comprehensively evaluating the density, velocity consistency, and reflection intensity distribution of the spatiotemporal vectors. For example, the probability of an object's existence can be calculated through a Sigmoid-based Classification Head or Softmax probability operation. That is, the probability that the location belongs to the object existence class is predicted as a real value between 0 and 1, and the closer the value is to 1, the more likely the location is to be an actual object.

[0108] The preset threshold probability can be preset to a value in the range of, for example, 0.7 to 0.9 and used as a reference value to exclude noise or non-objective reflection points.

[0109] The radar center analysis model can extract an image region based on projected position coordinates as a low-resolution ROI image by projecting the position coordinates of at least one first traffic object candidate onto the image coordinate system of the camera module.

[0110] The image coordinate system is a two-dimensional image space recognized by the camera module, and can be composed of horizontal (u) and vertical (v) coordinates in pixel units.

[0111] Projecting onto the image coordinate system of the camera module may be a process of converting the 3D position information (x, y, z) of the radar coordinate system into corresponding coordinates (u, v) in the image using the intrinsic parameters and extrinsic parameters (focal length, rotation matrix, projection matrix, etc.) of the camera module.

[0112] Extracting a low-resolution ROI image may be a process of cropping an image area of ​​a preset size (e.g., 64*64 to 128*128 pixels) centered on projected coordinates (u, v) and reducing the image area to normalize it for improved computational efficiency, thereby simplifying the shape information of the object.

[0113] The radar-centric analysis model can convert low-resolution ROI images into image embedding vectors through a lightweight convolutional network.

[0114] Lightweight convolutional networks may include a Depthwise Separable Convolution structure to improve computational efficiency. Depthwise Separable Convolution is a structure that reduces the amount of computation by separating existing convolution operations into channel-level depthwise convolution and 1x1 point convolution. Through this, the number of model parameters and FLOPs (Floating Point Operations) can be significantly reduced while maintaining the key features of the image (edges, contours, textures).

[0115] For example, an input low-resolution ROI image (e.g., 128*128 pixels) can be converted into a multi-stage feature map by passing through multiple convolutional layers and activation function layers. Each convolutional layer separates low-frequency and high-frequency components of the image through a filter and stabilizes the distribution of feature values ​​through batch normalization, thereby ensuring stable feature representation even under various lighting and noise conditions.

[0116] Subsequently, feature maps generated through multi-stage convolution operations can be averaged across spatial dimensions (Height, Width) through Adaptive Average Pooling operations and compressed into fixed-length feature vectors. Here, Adaptive Average Pooling may be an averaging-based downsampling operation designed to convert the size of the output feature map to a pre-set fixed size, regardless of the size of the input feature map. For example, if the input feature map is 32*32 in size and the output size is set to 1*1, Adaptive Average Pooling can calculate the average value of the entire area and compress it into a single value. For example, the feature maps generated through multi-stage convolution operations may be feature maps with 32 to 64 channels.

[0117] Compressed feature vectors can be converted into image embedding vectors by passing through a fully connected layer. Image embedding vectors may be the result of numerically representing the visual features (contours, color contrast, shape, and pattern) of low-resolution ROI images in a high-dimensional space. In other words, the generated image embedding vectors can include not only the morphological features of objects but also key visual information for identifying actual traffic objects, such as reflection patterns, contrast, and texture. For example, morphological features that are difficult to distinguish using radar alone, such as vehicle headlight reflections, pedestrian silhouettes, and the circular outlines of bicycles, can be quantitatively represented through image embedding vectors.

[0118] Therefore, a lightweight convolutional network can perform the role of computationally efficiently extracting visual features of objects from low-resolution ROI images and outputting image embedding vectors for a fusion operation (Fusion Layer) with radar feature vectors in a subsequent stage.

[0119] The radar-centric analysis model can generate fused feature vectors by combining image embedding vectors with radar feature vectors corresponding to spatiotemporal feature maps.

[0120] A fused feature vector can be an integrated vector formed by aligning radar feature vectors and image embedding vectors corresponding to the same object to the same dimension and then concatenating them at the channel level. In other words, a radar-centric analysis model can form a composite representation that complementarily reflects the information provided by the two modules by combining radar-based physical information and image-based visual information in parallel for the same spatial location.

[0121] The radar-centered analysis model can determine whether at least one first traffic object candidate corresponds to an actual traffic object by performing classification and regression operations on the fused feature vector.

[0122] The classification operation may be an operation to determine the probability of object existence and fine-grained categories based on fused feature vectors. To this end, the radar-centered analysis model can calculate probability distributions for multiple object classes by applying a softmax or sigmoid activation function in the fully connected layer.

[0123] Object classes can be established in a multi-level hierarchical structure according to object types that affect driving and signal control in a traffic environment. For example, object classes in a radar-centric analysis model may include primary and secondary classification schemes. For example, object classes for the primary level may include vehicles, pedestrians, two-wheelers, and non-static obstacles. For example, object classes for the secondary level may include, in the case of vehicles, sedan, SUV, van, bus, light truck, and heavy truck; in the case of two-wheeled vehicles, motorcycle, scooter, bicycle, and E-scooter; in the case of pedestrians, adult, child, elderly, and wheelchair (Assisted Mobility); and in the case of non-fixed obstacles, traffic sign, traffic light body, guardrail, road curb, static obstacle, and fallen object.

[0124] The radar-centered analysis model calculates independent probability values ​​for each object class and can determine the class with the highest probability as the object class.

[0125] For example, if a low-resolution ROI image is initially classified as a “Vehicle,” it can be further subdivided into a “Heavy Truck” or “Bus” through additional subheadings. This multi-layered classification structure can be implemented by adding a class-specific attention map (Channel Attention Map) to a lightweight convolutional network-based backbone (MobileNet-V3, EfficientNet-Lite, etc.) to enhance local visual features (window arrangement, body height, contour ratio, etc.) within the low-resolution ROI image.

[0126] Regression operations are operations for predicting spatial and kinematic properties of the same object, and can produce continuous parameters including the object's center coordinates (x, y), width, length, height, direction of travel (yaw), and velocity components (vx, vy). For example, a radar center analysis model can correct the position and size of the actual object by taking a fused feature vector as input and calculating the residual offset for the initial detection coordinates through a regression head based on a Multi-Layer Perceptron (MLP).

[0127] For example, the regression head has an MLP-based multilayer structure, and in the input layer, each component of the fused feature vector (position, velocity, RCS, image contour, etc.) is normalized and input, and in the hidden layer, non-linear relationships can be learned by applying a ReLU or GELU activation function. Then, in the output layer, each regression parameter is output as an independent node to produce individual continuous correction values.

[0128] For example, a radar-centered analysis model can determine an object as an actual traffic object if the probability of object existence calculated in the classification operation is greater than or equal to a preset threshold and the correction error calculated in the regression operation is within the allowable error.

[0129] The radar-centered analysis model can determine information about a first traffic object, including the location, speed, size, direction of travel, and object class of the first traffic object corresponding to an actual traffic object.

[0130] The position of the first traffic object can be expressed as (x, y) coordinates representing the object center point. The speed of the first traffic object can be expressed as the rate of change of position per unit time (vx, vy). The size of the first traffic object may include values ​​for width, length, and height, each predicted as a result of a regression operation, and may represent the occupied area of ​​the object on the actual road. The direction of travel of the first traffic object may be the yaw value expressed as the main movement vector of the traffic object or the angle of the vehicle body direction. The object class of the first traffic object is class information determined as a result of a classification operation, and may include identification values ​​for the class type for the first classification and the class type for the second classification, respectively.

[0131] Additionally, according to one embodiment, a radar-centric analysis model can be trained to predict the probability of existence, location, size, direction, speed, etc. of a traffic object using radar detection data and corresponding image data as training data. This training process can be performed using a supervised learning method, and the training data may consist of temporal multi-frame radar data collected from an actual road environment and camera image data at the same time.

[0132] For example, each image frame used as training data may include ground truth values ​​for the object's bounding box, object class (Class1, Class2), center coordinates (x, y), width, length, height, direction angle (yaw), and velocity vector (vx, vy) through a labeling tool. In this case, the labels for the ground truth values ​​may be time-series matched through Kalman correction or optical flow-based tracking to maintain continuity between frames.

[0133] For example, the total loss function can be set as a loss function based on four items: classification loss, regression loss, rate consistency loss, and temporal attention consistency loss.

[0134] The classification loss parameter is used to evaluate the probability of object existence and the classification accuracy of object classes, and it can be configured to minimize the difference between the predicted probability distribution and the actual object labels. The classification loss parameter is based on Cross-Entropy Loss and can correct for data imbalance by assigning weights to classes that appear relatively infrequently, such as pedestrians or bicycles. Additionally, Focal Loss may be applied to increase training focus on objects with low detection probabilities.

[0135] The regression loss component evaluates how well predicted position, size, and orientation values ​​match actual ground truth data. By measuring the difference between the center coordinates, width, length, height, and orientation angle predicted by the radar center analysis model and the actual measurements, the model can be trained to ensure that the object's bounding box accurately encloses the actual object. To achieve this, a smooth L1 loss, which smoothly mitigates positional errors, can be used, while the accuracy of object size or rotation angle matching is calculated using an Intersection over Union (IoU) based loss to improve spatial consistency.

[0136] The velocity consistency loss item is a constraint that ensures the predicted velocity change between frames is maintained within a physically natural range. For example, it can be trained to keep the difference in velocity between the previous frame and the current frame small so that the same object does not change velocity abruptly or is not predicted in the opposite direction in consecutive frames. Through this, the radar-centric analysis model can guarantee the continuity and physical validity of velocity estimation over time series.

[0137] The temporal attention consistency loss parameter can be a setting that restricts attention weights within a transformer-based encoding network from changing abruptly over time. In other words, by preventing the attention to the same object from shifting rapidly or disappearing from frame to frame, the stability of spatiotemporal feature learning can be ensured. The temporal attention consistency loss parameter can be configured by comparing the distribution of attention scores between consecutive frames and ensuring that the amount of change remains below a certain level.

[0138] The four loss components can be adjusted by assigning different relative weights depending on the model's learning objectives and data characteristics. For example, if object positional accuracy is considered more important, the weight of the regression loss can be increased, while if velocity continuity is emphasized, the weight of the velocity consistency loss can be set significantly. For instance, the total loss is set as the weighted sum of the four loss components, and the radar-centric analysis model can be trained to minimize the total loss.

[0139] In step S250, if the radar reliability index is maintained below a preset second threshold value for a preset second number of frames or more, the control server can determine information about the second traffic object through the image-centric analysis model based on the sensing information by activating the image-centric analysis model.

[0140] For example, the control server can ensure the stability of traffic object recognition by switching to an image-centric analysis model when the radar reliability index is maintained below a preset second threshold value (e.g., 0.4 or less) and for a preset second number of frames (e.g., 10 frames or more).

[0141] A preset second threshold value is a reference value for determining whether radar detection reliability has deteriorated to a level where it is difficult to maintain normal recognition performance, and can be set as a real value between 0 and 1. For example, a preset first threshold value for activating a radar-centric analysis model can be set to a value greater than the preset second threshold value for activating an image-centric analysis model. That is, the preset second threshold value is set to a value smaller than the first threshold value, and operation can be automatically switched to an image-centric analysis model when a decrease in radar sensitivity, external environmental influences such as rain or fog, or a hardware malfunction is detected.

[0142] The second preset number of frames is a temporal verification interval for determining whether the decrease in radar reliability is a continuous state rather than a temporary phenomenon, and, for example, when 5 frames of image data are collected per second, it can be set to an interval corresponding to 10 to 30 frames. In this way, if the radar reliability is maintained below a threshold value during the set interval, the control server can control the output of the radar-centric analysis model to be withheld and the image-centric analysis model to be activated in the main operation mode.

[0143] In addition, the control server can simultaneously ensure the efficiency and continuity of traffic object recognition by activating the image-centric analysis model while the radar reliability index remains below a second threshold, and then deactivating the image-centric analysis model and returning to the radar-centric analysis model when the radar reliability index recovers above a first threshold and is maintained for a preset first number of frames.

[0144] Here, the image-centric analysis model may be a deep learning-based analysis model that determines traffic objects based on image data received from a traffic information collection device by the control server. That is, when the reliability of radar detection data drops below a certain level, the control server can activate the image-centric analysis model as an alternative operation mode to recognize and correct traffic objects based on image data.

[0145] For example, an image-centric analysis model can generate a Multi-scale Feature Map based on image data and then convert the Multi-scale Feature Map into a BEV (Bird's Eye View) Feature Map in the BEV coordinate system. In this case, the Multi-scale Feature Map may include detailed visual information, such as image contours and boundaries, in the lower layers, and global pattern information, such as object shapes, background contrast, and size ratios within the image, in the upper layers. By integrating these features and converting them into a spatial representation in the BEV coordinate system, the road area can be reconstructed from a bird's-eye view. In other words, the BEV Feature Map may be a spatial feature map expressed by converting Multi-scale features extracted from input image data into a bird's-eye view BEV coordinate system. By aligning each location on the road in a bird's-eye view format, the BEV Feature Map can consistently represent the actual location of an object and its surrounding spatial relationships on a flat plane. For example, the positions of vehicles, pedestrians, lanes, traffic lights, etc., are repositioned into relative coordinate relationships on the actual road plane, allowing for accurate analysis of distances between objects, directions of movement, and lane alignment status. The BEV coordinate system is a planar coordinate system projected from an overhead view of the actual road environment, serving as a reference coordinate system for converting the 2D pixel coordinates of images captured by a camera module into absolute position coordinates on the road plane. The BEV coordinate system consists of forward (y) and lateral (x) axes, and image pixel coordinates can be converted into spatial distance units (in meters) and aligned to the BEV coordinate system. Through this, changes in the actual position of objects can be compared and analyzed based on the same standard, even between image frames captured from various viewpoints.

[0146] For example, an image-centric analysis model can determine information about a second traffic object by combining a radar hint map projected into a BEV coordinate system and a BEV feature map based on radar detection data.

[0147] A radar hint map may be represented as a weighted map that highlights areas with a high probability of object existence on an image-based feature map by projecting the distance, azimuth, and reflection intensity included in radar detection data to align with the BEV coordinate system. In other words, the radar hint map may be information that assists an image-centric analysis model in more accurately estimating the location of object candidates by establishing a correlation between radar module-based distance and velocity information and image-based visual information. For example, areas in the radar detection data where the magnitude of the reflection signal is detected to be larger than a preset size may be displayed on the image feature map with an attention weight higher than a preset weight, thereby improving the reliability of object detection.

[0148] According to one embodiment, the image-centric analysis model can determine each image frame collected during a preset second number of frames based on image data.

[0149] Here, a video frame is a video image of a single point in time captured continuously at regular time intervals by a camera module, and may be data in the form of a two-dimensional matrix containing color values ​​(R, G, B) and brightness values ​​(Luminance) for each pixel. That is, the video frame may be a still image on the time axis.

[0150] The image-centric analysis model can generate multiple image feature maps with different resolutions through a multi-stage convolutional neural network based on a pre-set second number of image frames. The image-centric analysis model can generate multi-resolution feature maps by integrating the multiple image feature maps through a Feature Pyramid Network (FPN).

[0151] In this case, a multi-stage convolutional neural network may include multiple convolution blocks. Each convolution block can be classified into low-level, mid-level, and high-level layers according to its hierarchical structure. By outputting feature maps with different resolutions and channel counts, each layer can simultaneously preserve the detailed contours of an object and global semantic information.

[0152] For example, the lower layer can perform operations at the original resolution of the input image or at half the resolution (1 / 1 to 1 / 2 scale). By repeatedly applying convolution operations using a 3x3 kernel and a 1-pixel stride (stride=1), the lower layer can extract low-level visual features, including gradients, edges, contours, textures, and color contrasts, within the local neighborhood of the pixels. For example, if the input image has a resolution of 1280x720, the output of the first convolution block can be generated as a feature map of the same resolution (1280x720) or a downsampled 640x360. In this case, the number of channels starts from the 3 RGB channels of the input image and can be expanded to 32 or 64 channels depending on the number of filters. Through such channel expansion, various low-level patterns, such as color, boundaries, and illumination changes, can be represented in parallel within the same resolution.

[0153] The intermediate layer takes feature maps output from the lower layer as input and can learn structural forms and sub-composition relationships within a wider spatial domain. The intermediate layer can learn contextual information from a wider area by reducing the spatial resolution to half or one-fourth of its original level (e.g., reducing 640*360 to 320*180 or 160*90) through convolution operations using a stride of 2 pixels or by applying max pooling. For example, each convolutional block in the intermediate layer can use a 3*3 or 5*5 kernel to recognize repeating patterns, symmetric structures, and inter-part relationships of objects, and the number of output channels for each block can be expanded to 128 or 256. In other words, the intermediate layer can secure richer representation capabilities by expanding the channel dimension instead of lowering the spatial resolution. The feature map output from the intermediate layer may include mid-level semantic features with structural continuity, such as road surfaces, vehicle fronts, pedestrian bodies, and lane dividers.

[0154] The upper layer takes the feature map output from the intermediate layer as input and can learn the overall shape and semantic context of an object by hierarchically accumulating multiple convolutional blocks. The upper layer applies convolutional operations with kernel sizes of 3x3 or 7x7 and repeatedly uses convolutional operations with a stride of 2 pixels to reduce spatial resolution to 1 / 8 or 1 / 16 of the original level (e.g., reducing 160x90 to 80x45 or 40x22). At this time, the number of channels is expanded to 512 or 1024, allowing for the inclusion of high-level semantic representations capable of responding to multiple object classes, background patterns, shadows, and lighting changes per single pixel. Consequently, while the output of the upper layer has low spatial resolution, it can improve global context and semantic differentiation capabilities. For example, through the upper layer, the positional relationships of pedestrians, vehicles, traffic lights, lanes, intersection markers, etc., within the same road section can be interpreted globally.

[0155] As a result, for a single image frame, each layer can generate multiple image feature maps with different resolutions (e.g., 1 / 2, 1 / 4, 1 / 8, 1 / 16 scale) and number of channels (64, 128, 256, 512 or more).

[0156] The FPN structure can improve object boundary discrimination and detection consistency compared to single-resolution feature maps by top-down fusion of abstract semantic information from the upper layer with detailed spatial information from the lower layer. As a result, the image-centric analysis model can generate multi-resolution feature maps that reduce feature imbalance between resolutions.

[0157] The image-centered analysis model can map to a BEV coordinate system of a preset resolution by back-projecting each pixel feature of a multi-resolution feature map into a ground plane coordinate system using camera intrinsic parameters and camera extrinsic parameters.

[0158] Specifically, camera intrinsic parameters may include the lens focal length, optical center coordinates (cx, cy), and distortion coefficient. The image center analysis model can calculate the direction in which a ray transmitted from a pixel is directed in actual 3D space for each pixel coordinate (u, v) of a multi-resolution feature map. For example, pixel coordinates (u, v) can be normalized by an intrinsic parameter matrix (K) and converted into coordinates (u', v') on the normalized image plane. This process is a step for converting the image coordinate system to the camera coordinate system, which can map the position of each pixel to a real-world reference direction vector.

[0159] The camera extrinsic parameters may include the camera's 3D position (translation: Tx, Ty, Tz) and a rotation matrix (R) for the tilt (orientation: roll, pitch, yaw). The image-centric analysis model can convert the corresponding point (x, y) of each pixel into absolute coordinates in the ground plane coordinate system by combining the direction vector transformed through the intrinsic parameters with the extrinsic parameters to calculate the intersection point where each pixel ray reaches the actual road plane (z=0). In this case, in a BEV transformation environment where the actual height of the road is assumed to be constant, the elevation (z value) in pixel units can be set to a fixed value (e.g., z=0).

[0160] Each transformed point (x, y) can be projected onto a grid cell of a preset BEV coordinate system. The BEV coordinate system can be defined as a 2D normalized grid structure that divides a range of forward distance (e.g., 0–60 m) and left / right distance (e.g., ±30 m) into equal intervals (e.g., 0.1 m–0.5 m resolution). Accordingly, each pixel is mapped to the BEV grid cell to which it belongs, and at this time, multiple pixel features projected onto the same grid can be aggregated into visual information corresponding to the same spatial interval.

[0161] The image-centric analysis model can generate a BEV feature map by aggregating pixel features projected onto the same grid through averaging operations.

[0162] An image-centric analysis model can aggregate features at each grid unit level in the BEV coordinate system by performing mean pooling or weighted average pooling on multiple pixel features projected onto the same BEV grid. For example, in the case of weighted average pooling, information loss can be minimized by applying at least one of the pixel reflectance, brightness value, or object probability within the image as a weight. Through this aggregation process, each BEV grid cell can be represented as a summarized feature vector that represents the visual information of that location. For instance, if multiple pixels constituting the roof of the same vehicle are projected at different image resolutions, all of these pixels may correspond to the same spatial location on the road (e.g., the grid where the vehicle's center is located). In other words, the feature values ​​of multiple pixels can be aggregated into the same BEV grid cell using mean pooling or weighted average pooling methods and integrated into a summarized feature vector representing that grid. The summary feature vector generated in this way can function as a vector representing the center position of the vehicle in the BEV coordinate system. In this way, the image-centered analysis model can generate a BEV feature map by combining the aggregated summary feature vectors of BEV grid units into a two-dimensional array form according to grid coordinates (x, y). The BEV feature map is a feature space representing objects, lanes, traffic signals, structures, etc. on the road from a bird's-eye view, which can remove perspective distortion from the image input and represent the distances and relative positions between objects as normalized planar coordinates.

[0163] Therefore, image-centric analysis models can improve the precision of object detection and location estimation by converting image inputs, whose size varies depending on perspective, into a spatial representation of uniform resolution.

[0164] The image-centric analysis model can generate a fused BEV feature map by fusing the BEV feature map and the radar hint map through cross-attention operations.

[0165] A radar hint map is a spatially weighted map that projects distance, azimuth, and reflection intensity information from radar detection data onto the BEV coordinate system and assigns high weights to locations with a high probability of object existence.

[0166] Cross-attention operations can be performed by setting each grid vector of the BEV feature map as a query vector and the corresponding grid vectors of the radar hint map as key and value vectors. At this time, the importance of the corresponding location on the radar hint map can be calculated as an attention weight through a dot product operation between the query vector and the key vector. By multiplying the calculated attention weight by the value vector, the visual characteristics of the corresponding grid can be dynamically corrected according to the radar-based signal reliability. That is, each grid within the BEV feature map can have its visual characteristics corrected by considering the reflected signal intensity, distance consistency, and azimuth angle changes of surrounding radar points. Through this, accurate object location estimation becomes possible even for distant objects that are difficult to identify in the image alone or in areas with non-uniform lighting.

[0167] For example, even if the outline of an object is unclear in an image due to brightness non-uniformity, if the radar hint map detects a strong reflection signal at the same location, the attention weight of that grid increases, making it more likely for the model to recognize that area as an object. Conversely, in the case of areas where the outline is clear in the image but the radar reflection signal is irregular, a low weight is assigned, which may lead to the area being identified as a noisy region. In this way, through a cross-attention-based fusion process, a fused BEV feature map—a high-dimensional representation that integrates shape information (Edge, Texture) of visual features with distance and velocity information of the radar signal—can be generated.

[0168] As a result, radar signal-based correction is applied even to distant objects or areas with non-uniform lighting that are difficult to identify in images alone, allowing for the stable estimation of the object's location, size, and probability of existence.

[0169] The image-centric analysis model can determine the object existence probability, center coordinate offset, size, direction of travel, speed, and object class at the BEV grid level based on the fused BEV feature map.

[0170] The image-centered analysis model can determine information about a second traffic object, including the location coordinates, speed, size, direction of travel, and object class of at least one second traffic object, by selecting a grid where the probability of object existence is greater than or equal to a preset threshold probability as the location coordinates of at least one second traffic object.

[0171] The image-centric analysis model includes a Multi-Head Output Layer and can perform classification and regression operations in parallel.

[0172] The image-centric analysis model can take a summary feature vector, i.e., a grid vector, for each grid of the fused BEV feature map as input and perform a classification operation to predict the probability of an object's existence. At this time, the multi-output head layer can calculate the probability of an object's existence (Logit value) through a fully connected layer based on each grid vector and convert it into a probability value by applying a softmax or sigmoid activation function. For example, if a grid of the BEV map is classified into two classes, “object present (1)” and “non-object (0)”, sigmoid activation is applied to produce a probability output (e.g., 0.87, 0.12, etc.). At this time, only grids where the probability of an object's existence exceeds a preset threshold probability (e.g., 0.7 or higher) can be set as targets for regression operations on location and velocity in subsequent steps, i.e., as traffic objects.

[0173] Through this structure, unnecessary computational load from background areas or noisy reflections can be removed, and learning focused on the object-centered area can be achieved.

[0174] For grids where the probability of an object's existence exceeds a preset threshold probability, the multi-output head layer can determine the position offset and size information of the object centered on that grid. The position offset can be a relative displacement amount (Δx, Δy) with respect to the grid's reference coordinates (xo, yo), and the BEV coordinates of the actual object center can be corrected in the form of (xo+Δx, yo+Δy). The size can be composed of width, length, and height. For example, for a car, it can be regressed in the form of (width, length, height) = (1.8, 4.5, 1.5) [m], and for a pedestrian, in the form of (width, length, height) = (0.6, 0.6, 1.7) [m]. These regression operations can be performed based on at least one of a multilayer perceptron or a 1*1 convolutional structure, and can be learned through at least one of a Smooth L1 loss function or a GIoU (Generalized Intersection over Union) based loss function.

[0175] The image-centric analysis model can analyze spatiotemporal patterns within the fused BEV feature map to determine the object's direction of movement and velocity vector. In this case, the direction of movement is set as the rotation angle (yaw) on the BEV plane, and the determined rotation angle value can be converted into the object's direction vector (cosθ, sinθ). The velocity is regressed as the motion components (vx, vy) on the BEV plane, and real-time estimation can be achieved by reflecting the rate of change in position between adjacent frames. For example, a vehicle can be output in the form (vx = 8.3 m / s, vy = 0.2 m / s), corresponding to an average driving speed of 30 km / h, while a pedestrian can be output as (vx = 1.2 m / s, vy = -0.3 m / s). This velocity regression is performed by reflecting the difference (Δt) between frames in the time-series input and attention-based time weights, and can maintain stable prediction values ​​for abnormal velocity changes (e.g., sudden stops, instantaneous reflections).

[0176] The image-centric analysis model can determine which type an object belongs to when its presence is confirmed at each grid unit. The object classes of the image-centric analysis model can be the same as the object classes of the radar-centric analysis model. That is, the object classes can include primary and secondary classification systems.

[0177] In this case, the classification operation is performed in the form of a probability distribution with softmax activation applied, and the sum of probabilities for each class can be normalized to 1. For example, in the first classification step, softmax activation can be applied to determine which of the four major classes—vehicle, pedestrian, two-wheeled vehicle, or non-fixed obstacle—it belongs to. For instance, the first classification result for a specific grid can be produced as {Vehicle: 0.70, Pedestrian: 0.20, Two-wheeled Vehicle: 0.10, Non-fixed Obstacle: 0.00}. In the second classification step, the detailed class corresponding to the superclass selected from the first classification result can be determined. For example, in the case of a grid classified as a vehicle, it can be further classified into one of the detailed classes such as a small passenger car, SUV, bus, or heavy truck. The output probability of each classification step can maintain the sum of class probabilities at 1 through softmax normalization. In addition, for example, Class-Balanced Loss can be applied during the training process to mitigate data imbalances between classes. By assigning relatively high training weights to detailed classes with low occurrence frequencies (e.g., bicycles, wheelchairs, etc.), weight adjustment can prevent training biased toward the majority of classes and improve subclassification accuracy.

[0178] The image-centered analysis model can integrate the aforementioned calculation results to determine the corresponding grid as a second traffic object, which is an actual traffic object, only when the probability of object existence is greater than or equal to a preset threshold probability and the regression prediction error is less than or equal to the allowable error range.

[0179] Here, the preset threshold probability is a reference value for determining the reliability of the existence of a detected object, and can be set to a real value in the range of, for example, 0.7 to 0.9. The tolerance range can be defined based on the Mean Squared Error of the position coordinates or the Intersection over Union (IoU) of the bounding box, and, for example, a match within an error range of 5% to 10% or less can be determined as a valid detection.

[0180] The image-centric analysis model may finally determine the grid satisfying the above conditions as the actual traffic object and define the said object as the second traffic object. Information regarding the second traffic object may include the object's center position, speed, size, direction of travel, and object class. The position of the third traffic object may be expressed as (x, y) coordinates representing the object's center point. The speed of the second traffic object may be expressed as the rate of change of position per unit time (vx, vy). The size of the second traffic object may include values ​​for width, length, and height, respectively, predicted by the result of a regression operation, and may represent the occupied area of ​​the object on the actual road. The direction of travel of the second traffic object may be the traffic object's main movement vector or a yaw value expressed as the vehicle body direction angle. The object class of the second traffic object is class information determined by the result of a classification operation, and may include identification values ​​for the class type for the first classification and the class type for the second classification, respectively.

[0181] This second traffic object information can be utilized in the signal control stage of the control server together with the first traffic object information of the radar-centered analysis model.

[0182] Additionally, according to one embodiment, an image-centric analysis model can be trained to predict the probability of existence, location, size, direction, speed, and object class of traffic objects using image data and radar hint maps as training data. This training process can be performed using a supervised learning method, and the training data may consist of Temporal Multi-Frame image data collected from an actual road environment and radar hint maps projected from radar detection results at the same time.

[0183] For example, each image frame used as training data may include ground truth values ​​for the object's bounding box, object class (Class1, Class2), center coordinates (x, y), width, length, height, direction of travel (yaw), and velocity vector through a labeling tool. In this case, the label of each image frame can be registered into a road plane-based BEV (Bird's Eye View) coordinate system using camera intrinsic parameters (focal length, optical center coordinates, distortion coefficient) and extrinsic parameters (installation position, attitude, tilt). The ground truth values ​​can be labeled to maintain continuity between frames through Kalman correction or optical flow-based object tracking for time-series registration between frames.

[0184] For example, the total loss function can be set as a loss function based on four items: classification loss, regression loss, attention matching loss, and BEV projection consistency loss.

[0185] The classification loss parameter is used to evaluate the accuracy of object presence probabilities and class predictions calculated at the BEV grid level, and it can be configured to minimize the difference between the predicted probability distribution and the actual object labels. The classification loss parameter is based on cross-entropy loss and can correct for data imbalance by assigning weights to classes that appear relatively infrequently, such as pedestrians or bicycles. Additionally, focal loss can be applied to increase training focus on objects with low detection probabilities.

[0186] The regression loss is an item used to evaluate how well the predicted position, size, and orientation values ​​match the actual ground truth data. The image-centered analysis model can be trained to minimize the difference between the actual measured values ​​and the predicted center coordinates, width, length, height, and yaw values ​​in the BEV coordinate system. To this end, a Smooth L1 loss, which smoothly mitigates position errors, can be used, and the accuracy of object size or rotation angle can be calculated through the Generalized Intersection over Union (GIoU) ​​loss to improve spatial accuracy on the BEV plane.

[0187] The attention matching loss parameter is designed to guide the Cross-Attention module within the image-centric analysis model to focus on object presence areas on the radar hint map during the training process. It can be configured to minimize the difference between the predicted attention weight distribution and the actual object presence probability map. Through this, the model is trained to increase visual focus on areas where strong reflection signals are observed in the radar hint map, thereby ensuring stable detection performance even in non-uniform lighting areas or distant regions.

[0188] The BEV projection consistency loss is a constraint that ensures image features projected across multiple frames maintain a temporally consistent position in the BEV coordinate system. In other words, the model can be trained to minimize the change in projection coordinates between frames so that the same object is not projected to different positions in consecutive frames. Through this, the model can simultaneously secure BEV spatial consistency over time and stability of object positions.

[0189] The four loss items can be adjusted by assigning different relative weights depending on the model's learning objective and data characteristics.

[0190] For example, if the accuracy of object presence determination is important, the weight of classification loss can be increased, and if the stability of BEV projection is important, the weight of projection consistency loss can be set high.

[0191] For example, the total loss is set as a weighted sum of four loss items, and the image-centric analysis model can be trained to minimize the total loss.

[0192] In step S260, the control server can determine traffic signal control information based on information about the first traffic object or information about the second traffic object.

[0193] For example, the control server can determine the traffic conditions of an intersection (vehicle queue, pedestrian occupancy rate, speed by vehicle type, collision risk, etc.) in real time based on information about a first traffic object or information about a second traffic object, and determine traffic signal control information corresponding thereto.

[0194] For example, if the radar reliability index is maintained above a preset first threshold value for a preset first number of frames or more, the control server can determine the traffic status of the intersection in real time based on information regarding the first traffic object and determine corresponding traffic signal control information. For example, if the radar reliability index is maintained below a preset second threshold value for a preset second number of frames or more, the control server can determine the traffic status of the intersection in real time based on information regarding the second traffic object and determine corresponding traffic signal control information.

[0195] For example, the control server can quantitatively analyze traffic flow within an intersection using the object's trajectory, speed, direction of travel, and class value based on information about the first traffic object or information about the second traffic object.

[0196] For example, if the radar reliability index is maintained above a preset first threshold value for a preset first number of frames or more, the control server can quantitatively analyze the traffic flow within the intersection using the object's trajectory, speed, direction of travel, and class value based on information about the first traffic object. For example, if the radar reliability index is maintained below a preset second threshold value for a preset second number of frames or more, the control server can quantitatively analyze the traffic flow within the intersection using the object's trajectory, speed, direction of travel, and class value based on information about the second traffic object.

[0197] For example, the trajectories of objects belonging to the vehicle class can be tracked to calculate the left-turn queue length (Q) or the straight-lane occupancy ratio (O), and for objects belonging to the pedestrian class, the pedestrian occupancy ratio (P) and the crosswalk occupancy time (Tp) can be calculated. In this case, Q can be calculated based on the average stopping distance and trajectory density of the vehicle, and P can be evaluated based on the number of frames continuously detected within a certain area.

[0198] In step S270, the control server can transmit traffic signal control information to the signal control device.

[0199] For example, traffic signal control information may include at least one of commands to adjust the green / red duration of left turn, straight, and pedestrian signals, commands to control the display of electronic information boards, and commands to transmit vehicle speed exceeding or garage limit warnings.

[0200] For example, the signal control device can control the signal pattern of an intersection in real time by operating a road guidance device based on received traffic signal control information.

[0201] For example, the control server can dynamically adjust the cycle and duration of traffic signals by comparing the calculated Q and P values ​​with preset thresholds. For example, if the queue length Q of a left-turn lane exceeds a threshold (e.g., 15 m or 5 vehicles), the green duration (G1) of the left-turn signal can be increased. For example, if the pedestrian occupancy P exceeds a threshold (e.g., 0.6 or higher), the red duration (Rv) of the vehicle signal can be increased. In this case, the signal timing adjustment can be progressively corrected by reflecting the rate of change of trajectory data (ΔQ / Δt, ΔP / Δt). Through this, the control server can perform adaptive signal control based on traffic density and pedestrian flow.

[0202] For example, the control server can calculate the actual driving speed of an object based on the speed component included in the selected object information and determine whether the speed is exceeded by comparing it with the allowable speed for each vehicle type. For example, if the allowable speed, which is set to 60 km / h for passenger cars and 50 km / h for heavy trucks, is exceeded, a speed violation event may be generated. The generated event may include {vehicle type, location, measured speed, allowable speed, exceedance rate, display text ID}, and the control server can send a command to the signal control device to display the text “Caution: Speeding” or “Speed ​​limit 50 km / h” on an electronic display board based on the event information.

[0203] For example, the control server can determine whether the direction of entry into a collision risk zone, such as a bridge, tunnel, or under a traffic light, matches the height included in the object's dimensions and the object's direction of travel. If the object's height exceeds the height limit (e.g., 4.0 m) and the entry vector matches the collision risk zone, an Overheight Warning may be generated. The Overheight Warning can provide the driver with a "Warning: Entry into Overheight Zone" message in real time via an electronic display or voice broadcasting device.

[0204] FIG. 3 is a flowchart illustrating the operation of activating an artificial intelligence model in a system that recognizes traffic objects and controls traffic signals using radar data and image data based on artificial intelligence according to one embodiment. The embodiment of FIG. 3 can be combined with various embodiments of the present disclosure.

[0205] Referring to FIG. 3, in step S301, the control server can receive sensing information from a traffic information collection device.

[0206] For example, a control server can receive metadata containing time information and device identification information from a traffic information collection device along with sensing information.

[0207] Sensing information may include radar detection data and image data.

[0208] In step S320, the control server can calculate the radar reliability index based on radar detection data.

[0209] For example, the control server can determine values ​​for average reflection intensity, Doppler consistency, distance-velocity physical consistency, and object count stability based on radar detection data.

[0210] For example, the control server can calculate the radar reliability index by weighting and summing the values ​​for average reflection intensity, Doppler consistency, distance-velocity physical consistency, and object count stability, respectively.

[0211] In step S303, the control server can perform the basic operation mode.

[0212] For example, the control server can activate a radar-centric analysis model as a default operating mode to determine information about the first traffic object based on sensing information.

[0213] In step S304, the control server can determine whether the radar reliability index is greater than or equal to a preset first threshold value.

[0214] The preset first threshold value can be set to 0.8 as a reference value for determining whether the radar reliability index maintains a stable detection state.

[0215] In step S305, if the radar reliability index is above a preset first threshold value, the control server can determine whether to maintain that state for above a preset first number of frames.

[0216] For example, if the radar reliability index is 0.8 or higher, the control server can determine whether the radar reliability index is maintained at 0.8 or higher for a preset first number of frames or more. For example, the preset first number of frames can be set to 10 frames.

[0217] In step S306, if the radar reliability index is maintained above a preset first threshold for a preset first number of frames or more, the control server may activate the radar center analysis model.

[0218] For example, if the currently active artificial intelligence model is a radar-centric analysis model, the control server can maintain the radar-centric analysis model.

[0219] For example, if the currently active AI model is a video-centric analysis model, the control server can disable the video-centric analysis model and enable the radar-centric analysis model.

[0220] In step S307, if the radar reliability index is less than a preset first threshold, the control server can determine whether the radar reliability index is less than a preset second threshold.

[0221] The preset second threshold value is a reference value for determining whether radar detection reliability has deteriorated to a level where it is difficult to maintain normal recognition performance, and can be set to 0.4.

[0222] In step S308, if the radar reliability index is below a preset second threshold value, the control server can determine whether the radar reliability index is maintained in that state for more than a preset second number of frames.

[0223] For example, if the radar reliability index is less than 0.4, the control server can determine whether the radar reliability index is maintained at less than 0.4 for a preset second number of frames or more. For example, the preset second number of frames can be set to 20 frames.

[0224] In step S309, if the radar reliability index is maintained below a preset second threshold value for more than a preset second number of frames, the control server may activate an image-centric analysis model.

[0225] For example, if the currently active AI model is a video-centric analysis model, the control server can maintain the video-centric analysis model.

[0226] For example, if the currently active AI model is a radar-centric analysis model, the control server can disable the radar-centric analysis model and enable the image-centric analysis model.

[0227] Additionally, according to one embodiment, the control server can dynamically adjust a preset first number for maintaining a radar-centered analysis model and a preset second number for activating an image-centered analysis model through a segment setting model using a neural network, by inputting environmental information and values ​​for a plurality of detection items and a plurality of image quality values ​​acquired during a preset measurement period.

[0228] Environmental information is a value representing the environmental conditions for a preset future time interval, and may include values ​​for temperature, humidity, rainfall, wind speed, atmospheric pressure, and cloud cover for each time period. For example, the environmental information may be weather information for a region corresponding to a preset road area obtained from an external server.

[0229] The preset measurement period may be a standard time interval for the control server to collect sensing information from the radar module and camera module to evaluate changes in radar reliability index and image quality. For example, the preset measurement period may be set in 5-minute or 10-minute intervals.

[0230] In this case, values ​​for multiple image quality parameters may be determined based on image frames. For example, the values ​​for multiple image quality parameters may include values ​​for contrast stability, boundary clarity, illumination uniformity, and inter-frame consistency.

[0231] Contrast stability is an indicator representing the degree to which the average contrast between consecutive frames remains constant, and it can be used to quantitatively evaluate the ability to distinguish the boundaries of objects within a video frame. For example, the contrast stability value may be a value between 0 and 1 obtained by calculating the standard deviation of the rate of change in contrast between video frame t and video frame t₀₁, and then converting it into a value between 0 and 1 through inverse normalization relative to the maximum rate of change. The smaller the variation in brightness difference between frames, the closer the stability is to 1, and the larger the variation, the closer the stability is to 0.

[0232] Edge Clarity is an indicator representing the sharpness and edge strength of edges detected within an image frame, and it can have a direct impact on object shape identification and segmentation accuracy. Edge Clarity can be expressed as a value between 0 and 1 by applying a Sobel or Canny filter to average the gradient magnitudes of the detected edges and then normalizing the result to the maximum gradient magnitude relative to the total pixels. The higher the average edge strength, the closer the clarity approaches 1, while the weaker or blurrier the edges, the closer the clarity approaches 0.

[0233] Luminance Uniformity is an indicator representing how uniform the brightness distribution is across an entire image, and it can be a value that quantitatively reflects the effects of lighting deviations or shadows. Luminance Uniformity can be a value between 0 and 1 obtained by calculating the variance of pixel brightness within an image frame and denormalizing it by dividing it by the maximum variance. The uniformity value approaches 1 as the deviation and uniformity of the brightness distribution decrease, while it approaches 0 as lighting imbalance becomes more severe.

[0234] Frame-to-Frame Consistency is an indicator representing the degree to which changes in pixel brightness or feature points between consecutive frames remain constant, and it can be used to evaluate the stability of the video signal and the reliability of video-based tracking. Frame-to-Frame Consistency can be calculated using the Structural Similarity Index (SSIM) between video frame t and video frame t₀₁, and the corresponding SSIM value can be used directly as a value between 0 and 1. The value approaches 1 as the similarity between video frames is high and noise fluctuations are low, while it approaches 0 as fluctuations between video frames are large and unstable.

[0235] For example, the control server can generate a time-series input vector by combining values ​​for multiple detection items and multiple image quality values ​​acquired during a preset measurement period in chronological order. That is, the time-series input vector can be configured to simultaneously reflect radar signal-based detection stability and image-based quality variability.

[0236] The interval setting model can be implemented as a deep learning-based neural network structure and, for example, may include an input normalization layer, a feature encoder layer, a state combining layer, and an active interval prediction layer.

[0237] The input normalization layer can generate a standardized time series vector with a mean of 0 and a variance of 1 by calculating the moving average (EMA) and standard deviation for each value included in the time series input vector. In this case, for example, if there are missing values, the input normalization layer can generate a normalized time series vector by replacing the missing values ​​with the moving average value over a preset number of frames (e.g., 5 frames).

[0238] The feature encoder layer may be a layer that learns changes in detection reliability and image quality over time and performs time-series encoding that simultaneously reflects short-term fluctuations and long-term trends.

[0239] For example, the feature encoder layer receives a normalized time-series vector as input from the input normalization layer and can perform Multi-Head Attention operations to learn the correlation between each time point. For instance, the importance of each time point is calculated through the attention operation, and this importance is expressed in the form of weights to highlight key sections of detection and image quality variations. For instance, the feature encoder layer can ensure learning stability and convergence by applying residual connections and layer normalization. For instance, the feature encoder layer can output a feature vector compressed along the time axis, split into two branches: a radar confidence feature vector and an image quality feature vector.

[0240] The state combining layer can combine the radar reliability feature vector and image quality feature vector extracted from the feature encoder layer with the environment state vector to generate an integrated representation vector that reflects the impact of environmental changes on reliability.

[0241] For example, a state concatenation layer can generate a single composite input vector by concatenating the radar reliability feature vector, the image quality feature vector, and the environment state vector. Here, the environment state vector can be generated through data preprocessing of environmental information. The environment state vector may include values ​​for temperature, humidity, rainfall, wind speed, atmospheric pressure, and cloud cover, respectively, for the time period corresponding to the next measurement period. Here, the time period corresponding to the next measurement period may be the time period corresponding to the measurement period following a preset measurement period among the multiple time periods included in the environmental information. The value for temperature may be in degrees Celsius. The value for humidity may be in relative humidity percentage. The value for rainfall represents the amount of rain that fell during a certain period and may be in millimeters. The value for wind speed may be in meters per second. The value for atmospheric pressure may be in hectopascals. The value for cloud cover represents the ratio of cloud cover to the entire sky and may be in decimals.

[0242] For example, the state combining layer can use a multilayer perceptron to perform a linear projection of a composite input vector and learn environment-specific weighting factors (e.g., first weights and second weights based on weather conditions). For example, under the constraint that the sum of the first and second weights is 1, each weight can be dynamically adjusted such that the first weight increases during bad weather or at night, and the second weight increases during daytime and clear weather. For example, the state combining layer can output the combined result as an integrated representation vector.

[0243] The active interval prediction layer can regressively predict the maintenance interval (first count) of the radar-centric analysis model and the activation interval (second count) of the image-centric analysis model using an integrated representation vector. For example, the active interval prediction layer includes separate regression heads for radar reliability and image reliability, respectively, and can calculate predicted values ​​through each head. In this case, the prediction result can be calculated in a direction that minimizes the cost function by considering the average reliability, variability, and cross-correlation per module. For example, the active interval prediction layer can output an output vector containing the maintenance interval (first count) of the radar-centric analysis model and the activation interval (second count) of the image-centric analysis model using an integrated representation vector. For example, the active interval prediction layer may include a regression head unit and a cost calculation unit. The regression head unit includes two independent regression output paths, and the first head can predict the maintenance interval (first count) of the radar-centric analysis model, while the second head can predict the activation interval (second count) of the image-centric analysis model. Each regression head may consist of a first-order fully connected layer, an activation function (ReLU or GELU), and a final output layer, and can output a scalar interval prediction value from the input integrated representation vector. The cost calculation unit may use a loss function to minimize the predicted first and second errors. The loss function includes the Mean Squared Error (MSE) between the predicted value and the actual interval value as a base term, but may additionally consider a cross-correlation term to prevent the prediction correlation between the radar module and the camera module from becoming excessively high or low. The cross-correlation term is calculated based on the covariance of the radar and image prediction results and can be weighted to simultaneously ensure prediction stability and mutual consistency.The active interval prediction layer can finally produce a two-dimensional output vector including the maintenance interval (first number) of the radar-centered analysis model and the active interval (second number) of the image-centered analysis model.

[0244] The segment setting model can perform initial training based on a training data set collected in advance prior to operation. For example, the control server can construct a training data set aligned by frame using radar item data and image data collected from multiple intersection environments. Each training data set may include radar reliability indicators (average reflection intensity, Doppler consistency, range-velocity consistency, etc.) and image quality indicators (contrast stability, boundary clarity, illumination uniformity, inter-frame consistency, etc.) corresponding to each frame.

[0245] For example, learning can be performed in a direction that minimizes the Mean Squared Error between the predicted activation interval results and the measured confidence-based reference interval values, and adaptive moment estimation-based optimization techniques (Adam) or mean-squared learning rate adjustment techniques (RMSProp) may be applied. Adaptive moment estimation-based optimization is a method that dynamically adjusts individual learning rates for each parameter, and can be an optimization technique designed to simultaneously improve the convergence speed and stability of Gradient Descent. This technique applies momentum to calculate the moving average of past gradients and automatically adjusts the learning rate for each parameter based on the result, thereby reducing oscillations and allowing it to reach the minimum loss point quickly. In other words, it can be a method that converges gradually by considering the trends of previous learning steps. Mean-squared learning rate adjustment is an optimization technique that mitigates learning instability by calculating the moving average of the squared gradient values ​​for each parameter and applying a small learning rate to parameters with large gradients and a large learning rate to parameters with small gradients. In other words, it is a method that automatically adjusts the learning rate by reflecting the magnitude of change in the recent interval.

[0246] For example, dropout and early stopping techniques can be applied to prevent overfitting.

[0247] For example, after training is complete, prediction accuracy, stability, and generalization performance are evaluated using validation data, and if the performance meets or exceeds the threshold, the trained model can be deployed to the production environment.

[0248] For example, the learned interval setting model can output the maintenance interval (first number) of the radar-centered analysis model and the activation interval (second number) of the image-centered analysis model based on the time-series input vector during the operation phase.

[0249] FIG. 4 is a block diagram showing the configuration of a control server according to one embodiment. The embodiment of FIG. 4 can be combined with various embodiments of the present disclosure.

[0250] As illustrated in FIG. 4, the control server (400) may include a processor (410), a communication unit (420), and a memory (430). However, not all components illustrated in FIG. 4 are essential components of the control server (400). The control server (400) may be implemented with more components than those illustrated in FIG. 4, or with fewer components than those illustrated in FIG. 4. For example, the control server (400) according to some embodiments may further include a user input interface (not shown), an output unit (not shown), etc., in addition to the processor (410), the communication unit (420), and the memory (430).

[0251] The processor (410) typically controls the overall operation of the control server (400). The processor (410) may be equipped with one or more processors to control other components included in the control server (400). For example, the processor (410) may control the communication unit (420) and the memory (430) overall by executing programs stored in the memory (430). Additionally, the processor (410) may perform the functions of the control server (400) described in FIGS. 1 to 3 by executing programs stored in the memory (430).

[0252] The communication unit (420) may include one or more components that enable the control server (400) to communicate with a traffic information collection device (not shown), a signal control device (not shown), a user terminal (not shown), and an external server (not shown). The other device (not shown) may be a computing device such as the control server (400) or a sensing device, but is not limited thereto. The communication unit (420) may receive user input from another electronic device or receive data stored in an external device from an external device via a network.

[0253] For example, the communication unit (420) can send and receive messages to establish a connection with at least one device. The communication unit (420) can transmit information generated by the processor (410) to at least one device connected to the server. The communication unit (420) can receive information from at least one device connected to the server. The communication unit (420) can transmit information related to the received information in response to the information received from at least one device.

[0254] The memory (430) can store a program for processing and controlling the processor (410). For example, the memory (430) can store information input to the server or information received from another device via a network. Additionally, the memory (430) can store data generated by the processor (410). The memory (430) can also store information input to or output from the control server (400).

[0255] The memory (430) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk.

[0256] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0257] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0258] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0259] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0260] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A traffic information collection device that collects information related to traffic conditions using a radar module and a camera module; a control server that recognizes traffic objects based on information collected from the traffic information collection device and generates traffic signal control information; The system includes a signal control device that controls a road guidance device based on the traffic signal control information, wherein the traffic information collection device acquires radar detection data and image data for a preset road area from the radar module and the camera module, and transmits sensing information including the radar detection data and the image data to the control server, and the control server determines values ​​for a plurality of detection items based on the radar detection data, calculates a radar reliability index based on the values ​​for the plurality of detection items, activates a radar-centered analysis model as a basic operating mode to determine information about a first traffic object based on the sensing information, maintains the activation of the radar-centered analysis model when the radar reliability index is maintained above a preset first threshold value for a preset first number of frames or more, and activates an image-centered analysis model when the radar reliability index is maintained below a preset second threshold value for a preset second number of frames or more, thereby determining information about a second traffic object through the image-centered analysis model based on the sensing information, wherein the preset first threshold value is a value greater than the preset second threshold value, and information about the first traffic object or the second The traffic signal control information is determined based on information about the traffic object, and the traffic signal control information is transmitted to the signal control device. The radar center analysis model determines at least one first traffic object candidate based on the radar point cloud included in the radar detection data, and thenInformation regarding the first traffic object is determined by extracting an image region corresponding to the location of at least one first traffic object candidate as a low-resolution ROI (Region of Interest) image, and the image-centric analysis model generates a multi-resolution feature map based on the image data, then converts the multi-resolution feature map into a BEV (Bird's Eye View) feature map in a BEV coordinate system, and determines information regarding the second traffic object by combining the radar hint map projected into the BEV coordinate system based on the radar detection data with the BEV feature map, and the radar-centric analysis model generates a radar point vector for each of the pre-set first number of frames by converting each radar point cloud collected during the pre-set first number of frames into planar coordinates using the distance and azimuth angle for each radar point, and the radar point vector includes a location vector and a radar feature vector for at least one radar point, and generates a spatiotemporal feature map reflecting speed changes and movement patterns between frames through a transformer-based encoding network based on the pre-set first number of radar point vectors, and the object existence probability based on the spatiotemporal feature map is pre A location coordinate greater than or equal to a set threshold probability is determined as the location coordinate of at least one first traffic object candidate, and by projecting the location coordinate of the at least one first traffic object candidate onto the image coordinate system of the camera module, an image region based on the projected location coordinate is extracted as the low-resolution ROI image, the low-resolution ROI image is converted into an image embedding vector through a lightweight convolutional neural network, the image embedding vector is combined with the radar feature vector corresponding to the spatiotemporal feature map to generate a fused feature vector, and classification and regression operations are performed on the fused feature vector,Determining whether the above-mentioned at least one first traffic object candidate corresponds to an actual traffic object, and determining information regarding the first traffic object including the location, speed, size, direction of travel, and object class of the first traffic object corresponding to the actual traffic object; the image-centric analysis model determines each image frame collected during the above-mentioned second number of frames based on the image data, generates a plurality of image feature maps having different resolutions through a multi-stage convolutional neural network based on the above-mentioned second number of image frames, generates a multi-resolution feature map by integrating the plurality of image feature maps through a Feature Pyramid Network (FPN), maps each pixel feature of the multi-resolution feature map to a BEV coordinate system of a pre-set resolution by back-projecting it into a ground plane coordinate system using camera intrinsic parameters and camera extrinsic parameters, and generates the BEV feature map by aggregating through an average operation on pixel features projected onto the same grid, and generates a fused BEV feature map by fusing the BEV feature map and the radar hint map through a cross-attention operation, and the fused BEV A system for determining an object existence probability, center coordinate offset, size, direction of travel, speed, and object class in BEV grid units based on a feature map, and determining information about a second traffic object including the location coordinates, speed, size, direction of travel, and object class of the at least one second traffic object by selecting a grid where the object existence probability is greater than or equal to a preset threshold probability as the location coordinates of the second traffic object. Claim 2 In claim 1, the control server adjusts the preset first number and the preset second number based on environmental information and values ​​for the plurality of detection items and values ​​for the plurality of image quality acquired during a preset measurement period, wherein the environmental information includes values ​​for temperature, humidity, rainfall, wind speed, atmospheric pressure, and cloud cover for each of the preset future time intervals by time period, and the values ​​for the plurality of image quality include values ​​for contrast stability, values ​​for boundary clarity, values ​​for illumination uniformity, and values ​​for inter-frame consistency. Claim 3 delete

Images