AI-Based Drone Vision?LiDAR Fusion Crowd Behavior Pattern Recognition and Risk Prediction System

Abstract

The present invention relates to an AI-based drone vision-LiDAR fusion crowd behavior pattern recognition and risk prediction system, and more specifically, to an AI-based drone vision-LiDAR fusion crowd behavior pattern recognition and risk prediction system that monitors crowds in real time using a drone and analyzes movement patterns and behavioral characteristics of multiple objects by utilizing AI-based image processing and 3D spatial sensor fusion techniques, thereby predicting potential dangerous situations caused by crowd density and issuing warnings in advance. The present invention may include a drone that acquires data regarding a controlled area from a vision sensor and a lidar sensor and acquires attitude data of the drone from an attitude sensor; an edge computing module mounted on the drone that generates metadata regarding the spatial location and movement of multiple objects within the controlled area by synchronizing and aligning the acquired data in time; a ground control server that receives the metadata from the edge computing module, generates an analysis result by analyzing the behavior of multiple objects, and calculates a risk level based thereon; and a dashboard that receives the metadata, analysis result, and risk level to perform visualization and alerting.

Description

Technology Field

[0001] The present invention relates to an AI-based drone vision-LiDAR fusion crowd behavior pattern recognition and risk prediction system, and more specifically, to an AI-based drone vision-LiDAR fusion crowd behavior pattern recognition and risk prediction system that monitors crowds within a controlled area in real time using a drone, and analyzes movement patterns and behavioral characteristics of multiple objects by utilizing AI-based image processing and 3D spatial sensor fusion techniques, thereby predicting potential dangerous situations caused by crowd density and issuing warnings in advance. Background Technology

[0003] Recently, unexpected safety accidents are frequently occurring at various events such as performances, festivals, and sports games where large crowds gather, as well as in busy downtown areas.

[0004] In particular, when crowds suddenly concentrate in bottleneck areas such as narrow alleys, entrances, and stairs, there is a high risk of serious stampede accidents due to crowd turbulence, where the flow of the crowd becomes irregular, or pressure increasing rapidly.

[0005] To prevent such accidents, it is necessary to observe and analyze complex behavioral patterns in real time, such as changes in crowd density, movement speed, direction of movement, and signs of abnormal flow.

[0006] However, conventional crowd surveillance and safety management measures have the following limitations.

[0007] First, fixed CCTV-based surveillance systems are prone to blind spots because their installation locations are limited.

[0008] There is a problem in that dense crowds cannot be detected in a timely manner due to insufficient monitoring of specific areas or spaces with height differences.

[0009] In addition, when using only general 2D image data, it is difficult to accurately calculate the actual density of crowds that appear overlapping on the screen because the actual distance (depth) between objects cannot be directly measured.

[0010] This can lead to errors in underestimating or overestimating crowd density, and there is also a problem where surveillance performance deteriorates at night, in backlit, or in bad weather conditions depending on lighting conditions.

[0011] Second, while simple drone video recording methods have the advantage of providing a wide field of view compared to fixed CCTVs, there is still a problem of insufficient depth information when utilizing only standard RGB camera footage.

[0012] Although image-based distance estimation techniques have been proposed, they have limitations such as low accuracy and significantly degraded detection performance depending on environmental factors like lighting conditions, smoke, or fog.

[0013] Third, the issue of personal data protection is also emerging as a significant challenge. Some existing crowd monitoring systems track individual objects based on facial recognition or personally identifiable information, but this raises serious concerns regarding privacy infringement and is restricted in use by various countries' personal data protection regulations (such as GDPR).

[0014] Accordingly, there is a demand for non-identification-based technologies capable of analyzing the movement paths of individual objects and collective behavioral characteristics within a crowd without directly identifying individuals.

[0015] In short, a new technology is needed that can simultaneously overcome the limitations of fixed 2D video surveillance systems and the requirements for personal information protection. Prior art literature

[0017] Republic of Korea Published Patent No. 10-2025-0105723 (Published July 9, 2025) Republic of Korea Published Patent No. 10-2025-0060686 (Published May 7, 2025) Republic of Korea Registered Patent No. 10-2579542 (Registered September 13, 2023) The problem to be solved

[0018] The present invention was devised to solve the problems of the prior art as described above. The objective of the present invention is to provide an AI-based drone vision-LiDAR fusion crowd behavior pattern recognition and risk prediction system that can monitor crowds in real time using a drone, and analyze movement patterns and behavioral characteristics of multiple objects without collecting personal identification information by utilizing AI-based image processing and 3D spatial sensor fusion techniques, thereby predicting potential dangerous situations caused by crowd density and warning of them in advance. means of solving the problem

[0020] The present invention for achieving the above objectives is,

[0021] It may include a drone that acquires data regarding a controlled area from a vision sensor and a lidar sensor and acquires attitude data of the drone from an attitude sensor; an edge computing module mounted on the drone that generates metadata regarding the spatial location and movement of multiple objects within the controlled area by synchronizing and aligning the acquired data in time; a ground control server that receives the metadata from the edge computing module, generates an analysis result by analyzing the behavior of multiple objects, and calculates a risk level based thereon; and a dashboard that receives the metadata, analysis result, and risk level to perform visualization and alerting.

[0022] Additionally, the edge computing module may include a sensor fusion module that calculates the spatial location of an object by fusing information obtained from the vision sensor, lidar sensor, and attitude sensor, and a non-identifying object tracking module that assigns an identifier to the object without personal identification information and generates the metadata by tracking the movement of the object.

[0023] Additionally, the ground control server may include a behavior pattern recognition module that recognizes the behavior pattern of an object based on location, movement, and distribution information of multiple objects included in the metadata, and a risk assessment module that evaluates changes in crowd density, movement anomalies, or dangerous situations according to the behavior pattern to calculate the risk level.

[0024] Additionally, the dashboard may include a drone image display unit that displays images received from the drone, a 3D situation display unit that displays the 3D distribution and density status of multiple objects, an event display unit that displays related events when a dangerous situation occurs, an alert module that provides an alarm according to the level of risk, a response action recommendation module that suggests measures to respond to the dangerous situation, and a drone control module that controls the movement or filming of the drone. Effects of the invention

[0026] According to the present invention, by utilizing a highly maneuverable drone platform, a wide area of ​​control can be rapidly observed, and there is an excellent effect of effectively eliminating blind spots that occur in fixed sensor-based systems.

[0027] The present invention has the effect of being able to more accurately recognize crowd density situations by precisely calculating the distance, density, and movement status between individual objects within a crowd through a 3D spatial sensor fusion technique that fuses a vision sensor, a lidar sensor, and a posture sensor.

[0028] In addition, by applying an edge computing-based non-identifiable object tracking structure, the present invention can reliably track multiple objects without exposing personally identifiable information and has the effect of minimizing analysis delays caused by communication delays by reducing the amount of data transmitted to the ground control server.

[0029] In addition, the present invention utilizes a time-series prediction model to predict not only the current risk state but also future changes in risk, thereby enabling preemptive warnings and response measures before a dangerous situation occurs, which has the effect of preventing crowd accidents.

[0030] In addition, the present invention has the effect of reducing the judgment burden on control personnel and improving response efficiency by intuitively visualizing risk levels through a dashboard and automatically providing alerts and response actions according to risk levels. Brief explanation of the drawing

[0032] FIG. 1 is a configuration diagram according to one embodiment of an AI-based drone vision-LiDAR fusion crowd behavior pattern recognition and risk prediction system according to the present invention. FIG. 2 is a configuration diagram of a drone in an AI-based drone vision-LiDAR fusion crowd behavior pattern recognition and risk prediction system according to the present invention. FIG. 3 is a configuration diagram of an edge computing module in an AI-based drone vision-LiDAR fusion crowd behavior pattern recognition and risk prediction system according to the present invention. FIG. 4 is a configuration diagram of a vision sensor in an AI-based drone vision-LiDAR fusion crowd behavior pattern recognition and risk prediction system according to the present invention. FIG. 5 is an example diagram schematically illustrating the object identification and tracking process according to the AI-based drone vision-LiDAR fusion crowd behavior pattern recognition and risk prediction system according to the present invention. FIG. 6 is a configuration diagram of a ground control server in an AI-based drone vision-LiDAR fusion crowd behavior pattern recognition and risk prediction system according to the present invention. FIG. 7 is a configuration diagram of a dashboard in an AI-based drone vision-LiDAR fusion crowd behavior pattern recognition and risk prediction system according to the present invention. FIG. 8 is a table showing an example of behavior classes classified by a risk assessment module of an AI-based drone vision-LiDAR fusion crowd behavior pattern recognition and risk prediction system according to the present invention. FIG. 9 is an example of the use of a dashboard in the AI-based drone vision-LiDAR fusion crowd behavior pattern recognition and risk prediction system according to the present invention. Specific details for implementing the invention

[0033] The embodiments of the present disclosure are illustrative for the purpose of explaining the technical concept of the present disclosure. The scope of rights according to the present disclosure is not limited to the embodiments presented below or the specific description thereof.

[0034] All technical and scientific terms used in this disclosure, unless otherwise defined, have the meaning generally understood by those skilled in the art to which this disclosure pertains. All terms used in this disclosure are selected for the purpose of further clarifying this disclosure and are not selected to limit the scope of the rights under this disclosure.

[0035] Expressions such as “comprising,” “comprising,” “having,” etc. used in this disclosure should be understood as open-ended terms implying the possibility of including other embodiments, unless otherwise stated in the phrase or sentence containing such expressions.

[0036] Unless otherwise stated, singular expressions described in this disclosure may include a plural meaning, and this applies likewise to singular expressions described in the claims.

[0037] Hereinafter, a preferred embodiment of the AI-based drone vision-LiDAR fusion crowd behavior pattern recognition and risk prediction system according to the present invention will be described in detail with reference to the attached drawings.

[0039] The present invention relates to an AI-based drone vision-LiDAR fusion crowd behavior pattern recognition and risk prediction system that can monitor crowds in real time using a drone, and analyze movement patterns and behavioral characteristics of multiple objects by utilizing AI-based image processing and 3D spatial sensor fusion techniques, thereby predicting potential dangerous situations caused by crowd density and providing advance warnings.

[0040] In the present invention, an object refers to an individual member of a crowd composed of multiple people, and the present invention aims to analyze the behavior of the crowd formed by the objects by individually tracking multiple objects and to predict crowd risk situations accordingly.

[0041] Referring to FIG. 1, the AI-based drone vision-LiDAR fusion crowd behavior pattern recognition and risk prediction system (50, hereinafter referred to as the 'system') according to the present invention may include a drone (100) that acquires data about a control target area from a vision sensor and a LiDAR sensor and acquires attitude data of the drone from an attitude sensor; an edge computing module (140) mounted on the drone (100) that generates metadata regarding the spatial location and movement of multiple objects within the control target area by synchronizing and aligning the acquired data in time; a ground control server (200) that receives the metadata from the edge computing module (140), generates an analysis result of analyzing the behavior of multiple objects, and calculates a risk level based thereon; and a dashboard (300) that receives the metadata, the analysis result, and the risk level to perform visualization and alerting.

[0043] Referring to FIG. 2, the drone (100) has an autonomous flight function using GPS, an indoor / outdoor position control system, a stabilization function including a gimbal and a vibration reduction device, and may be equipped with a flight control processor (101) for controlling it.

[0044] The detailed configuration of such flight control processor (101) is a known technology and is not claimed in the present invention, so a detailed description is omitted.

[0045] The above drone (100) can collect data about the controlled area by autonomously flying a set path in the controlled area or by flying around according to control commands from the dashboard (300).

[0046] More specifically, the drone (100) is equipped with a vision sensor (110) to acquire image data of a controlled area.

[0047] Referring to FIG. 4, the vision sensor (110) may include a visible light camera (111) and may further include a thermal imaging camera (112).

[0048] The thermal imaging camera (112) can supplement the image data obtained from the visible light camera (111) by detecting the body temperature of an object within the controlled area even in environments where visibility is reduced due to night or smoke, etc.

[0049] Referring again to FIG. 2, the drone (100) is equipped with a LiDAR sensor (120) to acquire three-dimensional spatial data of a controlled area.

[0050] In addition, the drone (100) is equipped with an attitude sensor (130) to acquire attitude data of the drone.

[0051] The attitude sensor (130) may be composed of a 6-axis inertial measurement unit (IMU).

[0053] Referring to FIG. 3, the edge computing module (140) may include a sensor fusion module (141) that fuses data obtained from the vision sensor (110), lidar sensor (120), and attitude sensor (130) to generate image-spatial fusion data for a controlled area and an object within the controlled area.

[0054] First, the sensor fusion module (141) can perform PTP (Precision Time Protocol, IEEE 1588) based time synchronization based on the GPS PPS (Pulse Per Second) signal of the drone (100) to accurately match the acquired data of the vision sensor (110) and the lidar sensor (120) that generate data at different cycles, and the accurate matching may mean that data at the same or adjacent generation times correspond to each other.

[0055] Additionally, the sensor fusion module (141) estimates the 6-degree-of-freedom attitude and velocity of the drone (100) through visual-inertial odometry (VIO) that combines the image data of the vision sensor (110) and the attitude data of the attitude sensor (130), and performs real-time drone attitude tracking by correcting it using an extended Kalman filter (EKF). The correction may be intended to reduce errors occurring in the process of estimating the position, attitude, and velocity of the drone by using drone movement information from a previous time point together with current sensor measurements.

[0056] The sensor fusion module (141) can use this drone attitude tracking information to perform error correction of the spatial data of the LiDAR sensor (120) caused by vibration or rotation occurring during the movement of the drone (100), and the error correction may be intended to reduce the positional error of the spatial data that occurs due to changes in the attitude of the drone.

[0057] Additionally, the sensor fusion module (141) can generate a depth map by projecting the spatial data of the aligned LiDAR sensor (120) onto the image data of the vision sensor (110) to assign a depth value to each pixel, and generate image-spatial fusion data based thereon, and the image-spatial fusion data may be data in which a depth value according to the depth map corresponds to each pixel of the image data.

[0058] The above image-spatial fusion data can accurately identify perspective relationships between objects compared to existing image data, which has an advantageous effect for object recognition and distance determination.

[0059] The sensor fusion module (141) can generate image-space-thermal fusion data by additionally combining thermal image data with image-space fusion data, and the image-space-thermal fusion data may be data that combines image data, space data, and thermal image data together.

[0060] The sensor fusion module (141) can perform dynamic weighted sensor fusion by estimating the reliability of the vision sensor (110) and the lidar sensor (120) using a Bayesian filter, for example, giving higher weight to the thermal image data of the thermal camera (112) and the spatial data of the lidar sensor (120) at night when the illumination is low, and giving weight to the image data of the visible light camera (111) and the spatial data of the lidar sensor (120) during the day.

[0061] Furthermore, the sensor fusion module (141) can correct drift occurring between the vision sensor (110) and the lidar sensor (120) during operation through online calibration using environmental feature points such as unique terrain features or building corners.

[0062] Accordingly, the sensor fusion module (141) can reliably generate aligned image-spatial fusion data for a controlled area and an object within the controlled area, and the aligned image-spatial fusion data may be data that combines image and spatial information that corresponds to each other in time and corrects errors due to changes in the drone's attitude.

[0064] Meanwhile, the edge computing module (140) may include a non-identifying object tracking module (142) that assigns a unique identifier to an object without personal identification information and generates metadata by tracking the movement of the object, and the metadata may include the object's location, direction of movement, speed, and ID information for object identification.

[0065] The above non-identifiable object tracking module (142) can detect objects by performing deep learning-based object detection based on image-spatial fusion data generated by the sensor fusion module (141), and can set a unique identifier (ID) for the detected objects.

[0066] At this time, the non-identifiable object tracking module (142) utilizes object detection results based on image-spatial fusion data as a basis, but depending on environmental conditions, it may refer to or mutually complement the object detection results performed using image data alone or spatial data alone.

[0067] In other words, the reliability of object detection can be improved by cross-checking whether an object detected in image data corresponds to spatial data.

[0068] In addition, even in cases where objects are difficult to detect in image data due to insufficient lighting, occlusion, or limited field of view, the position information of the object detected in the spatial data is reflected in the image-spatial fusion data according to the geometric alignment relationship between the image data and the spatial data, thereby enabling the recognition and tracking of the object.

[0069] Accordingly, missing object detection can be minimized even under various environmental conditions.

[0070] FIG. 5 is an example diagram schematically illustrating the object identification and tracking process performed by the non-identifiable object tracking module (142).

[0071] Referring to Fig. 5, multiple objects within the controlled area are detected based on image-spatial fusion data, and each object is displayed in the form of a bounding box and can be distinguished by being assigned a unique identifier (ID) without personal identification information.

[0072] The bounding box above is intended to visually represent the position and movement status of an object, and the position, movement direction, and speed information of each object can be continuously updated by the non-identifiable object tracking module (142) to generate metadata.

[0073] In addition, the above-mentioned non-identification object tracking module (142) can perform non-identification processing on image data prior to object tracking to protect personal privacy.

[0074] Specifically, the non-identifiable object tracking module (142) applies a real-time face detector to image data obtained from a visible light camera (111) and, when personal information such as a person's face or a car license plate is recognized, blurs the corresponding area so that information capable of personal identification is not exposed to the outside.

[0075] At this time, the de-identification object tracking module (142) is configured such that the image data used for object tracking is processed only temporarily in the memory area of ​​the edge computing module (140) mounted on the drone (100), and after the de-identification process is completed, the original image data is immediately overwritten and blocked from storage or external transmission.

[0076] In addition, the unique identifier (ID) set by the non-identifying object tracking module (142) is a random number-based anonymous identifier that is not logically linked to an individual's identity, facial information, or biometric information, and can be configured to be maintained only during the session for object tracking and automatically destroyed when the session ends.

[0077] Accordingly, the unique identifier (ID) above functions as a track key that enables the tracking of the continuous movement of an object, while not being used to re-identify a specific individual.

[0078] Accordingly, personal identification information included in the original video is not leaked outside the edge computing module (140), and the non-identifiable object tracking module (142) can structurally guarantee personal privacy.

[0079] Accordingly, no personal information remains in the video data, and subsequently, when extracting features for object tracking at the ground control server (200), only features other than personal information are used.

[0080] Accordingly, the present invention has the effect of reducing the amount of communication with the ground control server and minimizing analysis delays caused by communication delays by performing object detection and tracking in advance on an edge computing module mounted on the drone, rather than a server-centric centralized processing method.

[0081] Meanwhile, multiple drones (100) may be deployed in a controlled area, and each drone (100) may be configured to be equipped with at least some of a visible light camera (111), a thermal imaging camera (112), and a lidar sensor (120).

[0082] In one embodiment, the edge computing module (140) of each drone (100) can receive acquired data from other drones (100) via wireless communication and perform logically integrated sensor fusion processing.

[0084] Referring to FIG. 6, the ground control server (200) may include a behavior pattern recognition module (210) that recognizes the behavior pattern of an object based on location, movement, and distribution information of a plurality of objects included in metadata received from the edge computing module (140).

[0085] The above behavior pattern recognition module (210) analyzes the movement trajectories and behavior patterns of multiple objects using metadata that is continuously received over time.

[0086] Specifically, the behavior pattern recognition module (210) can perform a data association process that links the detection of an object detected in each frame with the previously managed object tracking state.

[0087] To this end, the behavior pattern recognition module (210) can predict the next location based on a Kalman filter for each object currently being tracked, calculate the residual with the newly detected observation, and then perform minimum cost matching using a Hungarian algorithm.

[0088] At this time, each of the above objects is defined to have a state vector including a 3D position, velocity, and acceleration, and the Kalman filter can predict the position of the object at the next time point based on the state vector of the previous time point and update the state vector using the error with the observed position.

[0089] In addition, the cost function for inter-frame object linkage can be calculated by weighted combination of multiple factors including the spatial distance between the predicted position and the observed position, the similarity between the external features of the objects, and the difference between the prediction speed and the observation speed, and the Hungarian algorithm can determine the correspondence relationship between objects based on the cost function so that the total cost is minimized.

[0090] In one embodiment, the behavior pattern recognition module (210) can perform object tracking by first predicting where an object being tracked in the previous frame will move in the next frame, and selecting the most appropriate object among newly detected objects based on the prediction result.

[0091] Specifically, the behavior pattern recognition module (210) can calculate the expected location of the object in the next frame based on the object's position, movement speed, and movement direction in the previous frame using a Kalman filter. This is similar to how a person expects "it will be near here next time" by observing the direction and speed of their walking.

[0092] Meanwhile, when multiple objects are newly detected in the next frame, the behavior pattern recognition module (210) can calculate a cost value for each detected object based on how close each detected object is to the predicted location, how similar its appearance is to the previous object, and whether its direction of movement and speed are naturally connected. At this time, the cost value is an indicator representing the probability that the detected object is identical to the object in the previous frame, and can be set lower as the probability increases.

[0093] The Hungarian algorithm described above utilizes the cost value to map objects to one another such that the total cost is minimized among the correspondence relationships between multiple previous objects and multiple detected objects. In other words, the Hungarian algorithm ensures that the 'pair of detected objects most plausibly connected to the previous object' is selected in the most natural way overall.

[0094] Accordingly, even in situations where multiple objects move simultaneously or move in close proximity to each other, the behavior pattern recognition module (210) can stably maintain continuous tracking of the same object by combining movement prediction by the Kalman filter and correspondence judgment by the Hungarian algorithm.

[0095] At this time, the behavior pattern recognition module (210) can calculate the matching cost by comprehensively considering the positional distance between objects, object size similarity, visual feature similarity, etc., and can determine that they are the same object if the cost is within a threshold.

[0096] If there is no existing trace object corresponding to a specific observation, a new trace can be started by assigning a new identifier; conversely, trace objects that are not observed for a certain period of time or longer can be removed from the trace list.

[0097] Accordingly, continuous identifiers per object can be stably maintained even in crowd environments where appearances and disappearances are frequent.

[0098] In addition, the behavior pattern recognition module (210) can improve tracking accuracy by using multimodal features that combine the physical state and visual appearance features of an object.

[0099] For example, the association between objects in a hybrid feature space can be calculated by combining the object's 3D position coordinates and instantaneous velocity vector with the appearance feature vector extracted from a deep learning-based re-identification (Re-ID) model.

[0100] Accordingly, the same object can be reliably reconnected even when temporarily occluded or reappearing after leaving the controlled area.

[0101] Meanwhile, the ground control server (200) may include a risk assessment module (220) that evaluates changes in crowd density, abnormal movement, or dangerous situations according to the behavioral pattern and calculates the risk level.

[0102] The above risk assessment module (220) can classify the behavior of objects in a crowd into a plurality of predefined behavior classes based on the results of the behavior pattern recognition.

[0103] For example, referring to FIG. 7, the behavior class may include, but is not limited to, at least one of normal walking with a constant speed and directionality, stopping or loitering by staying in a specific area for a long time, rapid acceleration of movement at more than twice the speed of normal walking, falling where the height (Z-axis) of the object is rapidly lowered, counter-movement where the crowd moves in the opposite direction to the main direction of movement, and abnormal gathering where the density rapidly increases in a narrow area.

[0104] In addition, to prepare for new behavioral patterns that are not previously defined, the risk assessment module (220) can perform an anomaly detection mode.

[0105] For example, using an autoencoder model trained on normal walking behavior patterns among behavior classes, the distribution of object behavior indicators over a certain time interval can be input, and the reconstruction error can be calculated.

[0106] The above risk assessment module (220) can determine that if the reconstruction error exceeds a threshold value, the crowd behavior pattern at that time is abnormal and process it as an abnormal behavior detection event.

[0107] The above risk assessment module (220) can calculate the risk level by combining multiple risk indicators to express the safety status of the entire crowd as a single quantitative indicator.

[0108] The above risk indicator may include at least one of density representing the number of people per unit area, velocity deviation representing the dispersion of object movement speeds, turbulence representing the disorder of crowd movement, backflow ratio representing the proportion of objects moving against the main direction of movement, and pressure gradient representing the degree of physical pressure change due to crowd density.

[0109] The above density can be calculated by estimating the number of people per m² by assuming the average occupancy area of ​​people from image-spatial fusion data.

[0110] In a static flow, everyone moves at a similar speed, but in a chaotic situation, speed variations increase as some people stop and others run; therefore, a high speed variation value can be interpreted as instability in the crowd flow.

[0111] The above turbulence diagram can be calculated from the vorticity of the velocity field or the relative velocity between adjacent objects by modeling the crowd as a fluid flow due to differences in the direction of movement of objects, sudden changes in direction, or vortex phenomena in the crowd flow.

[0112] Since the risk of collision and confusion increases as the number of people pushing through in the opposite direction increases within a crowd walking in one direction, the above-mentioned backflow ratio can be calculated by determining the main flow direction of the crowd and the ratio of the number of objects moving in a direction that differs from it by more than a predetermined angle, and an anomaly can be determined by setting a threshold value.

[0113] The above pressure gradient can be calculated as the divergence of the product of population density and velocity, and since an area with a large pressure gradient implies a strong pushing force toward the surroundings, it can be determined that there is a potential risk of crowds being pushed over or falling.

[0114] The above risk assessment module (220) can calculate a comprehensive risk index by converting the above risk indicators into a standardized scale, multiplying them by weights, and summing them.

[0115] At this time, the above weights can be set by analyzing the accident contribution of each risk indicator from past accident case data.

[0116] The summed risk index can be normalized to the previously collected minimum / maximum values ​​and displayed on a scale from 0 to 100 points.

[0117] Meanwhile, the above-mentioned risk assessment module (220) can perform a time series prediction model to predict not only the risk level at the current point in time but also the change in risk level after several seconds to tens of seconds.

[0118] The above time series prediction model may be a recurrent neural network-based model for learning the temporal movement patterns of a crowd, and may include, as an example, a Long Short-Term Memory (LSTM) network, but is not limited thereto.

[0119] Accordingly, the system can detect in advance situations where the risk level is likely to rise rapidly in a short period of time, even when the current risk index is below a threshold, and perform preemptive warnings or response measures.

[0120] The above risk indicators are each set to reflect the different occurrence mechanisms of crowd accidents and are intended to evaluate static risk due to personnel density within a unit area, dynamic instability risk due to movement imbalance between individual objects within the crowd, mass turbulence risk due to the disruption of movement flow, collision risk due to collision with the main direction of movement, and physical overturning risk due to crowd compression, respectively.

[0121] Accordingly, the risk assessment module (220) can more precisely determine the crowd risk state prior to an accident by combining multiple risk indicators that exhibit different risk characteristics, rather than relying on a single indicator.

[0122] In one embodiment, when a bottleneck occurs near a passage or entrance of a controlled area, the risk assessment module (220) can spatially analyze the pattern of disorder in crowd movement spreading to surrounding areas centered on the point where the bottleneck occurred, using image-spatial fusion data acquired from the drone (100).

[0123] Specifically, the risk assessment module (220) can calculate a velocity deviation representing the dispersion of object movement speeds, a turbulence calculated from the vorticity of the velocity field modeling the crowd as a fluid flow, and a backflow ratio calculated as the ratio of the number of objects moving in a direction that differs from the main flow direction of the crowd by more than a predetermined angle, using metadata in which the location, movement direction, and velocity information of each object is continuously updated. The risk assessment module (220) can track areas where the velocity deviation, turbulence, and backflow ratio increase in the vicinity of a bottleneck point in chronological order. In this case, it can identify a propagation path in which the unstable state of the crowd flow propagates to adjacent areas according to the direction in which the location of the area changes, and calculate a propagation speed according to the amount of change in the location of the area and the time taken. Accordingly, the risk assessment module (220) can generate a bottleneck phenomenon propagation map including the propagation path and the propagation speed.

[0124] Meanwhile, within the controlled area, there may already exist high-density areas where the static risk is assessed as high due to a high number of people per unit area, and such high-density areas are prone to crowd compression and can be classified as high-risk sections with a high probability of accidents.

[0125] The above risk assessment module (220) can use the above bottleneck propagation map to calculate the propagation time expected to be required for the disorderly state of crowd movement occurring at the bottleneck point to reach the high-density area classified as the high-risk section.

[0126] At this time, if the above disordered state reaches the above high-risk zone, the probability of collision in the crowd compression state increases rapidly, and it may be judged as an imminent accident state with a high probability of leading to mass overturning or chain accidents.

[0127] Accordingly, the risk assessment module (220) may be configured to assess the area as a section with a very high probability of accidents and provide a preemptive early warning to the manager, even before a crowd collapse or stampede accident actually occurs in the high-density area, when the propagation time is predicted to be less than or equal to a preset threshold time.

[0129] Referring to FIGS. 8 and 9, the dashboard (300) may include a control interface that visually provides metadata, analysis results, and risk levels received from the ground control server (200), and supports alarms and response measures.

[0130] Specifically, the dashboard (300) may include a drone image display unit (310) that displays image data received from the drone (100).

[0131] At this time, the video data displayed through the drone video display unit (310) is a video that has been de-identified by the de-identified object tracking module (142), and information that can identify an individual, such as a face, is not exposed.

[0132] Additionally, the dashboard (300) may include a three-dimensional situation display unit (320) that displays the three-dimensional distribution and density status of multiple objects.

[0133] The above three-dimensional situation display unit (320) can divide the control target area into a plurality of grid cells and visualize it in the form of a heat map based on the density and risk calculated for each grid cell.

[0134] At this time, high-risk areas can be distinguished and displayed using a relatively high-risk color (e.g., red), and low-risk areas using a low-risk color (e.g., green), and the heatmap can be updated in real time based on data received in real time.

[0135] Additionally, the above dashboard (300) may include an event display unit (330) that displays a relevant event when a dangerous situation occurs.

[0136] The above event display unit (330) can display at least one of an abnormal behavior detection event, a risk surge event, or a predicted risk threshold exceeding event transmitted from the risk assessment module (220).

[0137] Additionally, the dashboard (300) may include an alert module (340) that provides an alert according to the risk level and a response action recommendation module (350) that suggests measures to respond to the dangerous situation.

[0138] The above notification module (340) can provide an alert through multiple notification means, such as screen display, warning sound, and voice guidance, when a threshold value set in stages such as interest, caution, alert, and severe is exceeded according to the risk index, and the multiple notification means can be configured to operate automatically in conjunction with the above notification module (340).

[0139] Additionally, the above notification module (340) may be configured to preemptively provide an alert raised by one level, even if the current risk level is low, when it is determined that the risk index predicted by the time series prediction model will exceed a threshold.

[0140] In addition, the above response action recommendation module (350) can provide different response policies depending on the level of risk.

[0141] For example, in the interest phase, general monitoring and data logging are performed, and in the caution phase, notifications can be provided to control personnel or the intensity of drone surveillance can be increased.

[0142] In the alert phase, it transmits warning broadcasts and requests the dispatch of relevant agencies, such as security personnel or the police, and in the severe phase, it can coordinate with local or national disaster response systems.

[0143] In addition, the above dashboard (300) may include a drone control module (360) that controls the movement or filming of the above drone (100).

[0144] The above drone control module (360) can be configured to remotely control the position, flight path, or shooting direction of the drone (100) within the control target area.

[0146] In the present invention, the network may be a wired communication network or a wireless communication network. For example, the wireless communication network may be a wireless communication network using LTE (Long Term Evolution), 5G (5th Generation) and WiFi, etc., or a wired communication network such as Ethernet, xDSL (ADSL, VDSL), HFC (Hybrid Fiber Coaxial Cable), FTTC (Fiber to The Curb), FTTH (Fiber to The Home), etc.

[0147] The drone (100), ground control server (200), and dashboard (300) described in this specification refer to devices capable of transmitting and receiving various data via a network according to user operation. Such a network can perform voice or data communication.

[0148] The ground control server (200) and dashboard (300) described in this specification for this purpose may be equipped with a memory for storing programs and protocols for transmitting and receiving data with the drone (100), a microprocessor for executing various programs to perform calculations and control, etc.

[0150] Accordingly, according to the AI-based drone vision-LiDAR fusion crowd behavior pattern recognition and risk prediction system (50) of the present invention as described above, a wide area of ​​control target can be rapidly observed by utilizing a drone platform with excellent maneuverability, and blind spots occurring in fixed sensor-based systems can be effectively eliminated. Furthermore, through a 3D spatial sensor fusion technique that fuses vision sensors, LiDAR sensors, and attitude sensors, the distance, density, and movement status between individual objects within a crowd can be precisely calculated, thereby allowing for more accurate recognition of crowd density situations. Additionally, by applying an edge computing-based non-identifiable object tracking structure, multiple objects can be stably tracked without exposing personal identification information. Furthermore, the amount of data transmitted to the ground control server can be reduced, thereby minimizing analysis delays caused by communication delays. Moreover, by using a time-series prediction model to predict not only the current risk state but also future risk changes, preemptive warnings and response measures can be performed before a dangerous situation occurs, thereby preventing crowd accidents. Additionally, by intuitively visualizing the risk level through a dashboard and automatically providing warnings and response actions according to the risk level, the burden of judgment on control personnel is reduced and response efficiency is improved. It has various advantages.

[0152] The system described above may be implemented as hardware components, software components, and / or a combination of hardware and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. The processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0153] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer devices and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0154] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory.

[0155] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described device, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0157] The above-described embodiments are explained with respect to the most preferred examples of the present invention, but are not limited to the above embodiments, and it is obvious to those skilled in the art that various modifications are possible within the scope of the technical spirit of the present invention. Explanation of the symbols

[0159] 50: AI-based Drone Vision-LiDAR Fusion Crowd Behavior Pattern Recognition and Risk Prediction System 100 : Drone 101: Flight Control Processor 110: Vision sensor 111 : Visible light camera 112 : Thermal imaging camera 120: LiDAR sensor 130: Attitude sensor 140: Edge computing module 141: Sensor Fusion Module 142 : Non-identifiable object tracking module 200 : Ground control server 210: Behavior Pattern Recognition Module 220 : Risk Assessment Module 300 : Dashboard 310: Drone video display unit 320 : 3D situation display 330 : Event display 340: Notification Module 350: Response Action Recommendation Module 360: Drone Control Module

Claims

Claim 1 A plurality of drones that acquire data regarding a controlled area from vision sensors and LiDAR sensors and acquire attitude data of the drones from attitude sensors; an edge computing module mounted on each of the drones that generates metadata including location and movement information of multiple objects within the controlled area by synchronizing the acquired data in time, and performs logically integrated sensor fusion processing by receiving data acquired from other drones; an edge computing module that receives the metadata from the edge computing module and calculates a velocity deviation representing the dispersion of object movement speeds based on the location, movement direction, and velocity information of each object included in the metadata, a turbulence calculated from the vorticity of a velocity field modeling a crowd as a fluid flow, and a backflow ratio calculated as the ratio of the number of objects moving in a direction differing by more than a predetermined angle from the main flow direction of the crowd; and generates a bottleneck propagation map including the propagation path and propagation speed by which the unstable state of crowd flow propagates to adjacent areas by tracking the area where the velocity deviation, turbulence, and backflow ratio increase near the bottleneck point in chronological order, and using the bottleneck propagation map, the disorderly state of crowd movement occurring at the bottleneck point is classified as a high-risk section AI-based drone vision-LiDAR fusion crowd behavior pattern recognition and risk prediction system comprising: a ground control server including a risk assessment module that calculates the propagation time expected to be required to reach a high-density area and evaluates the high-density area as a section with a very high probability of accidents if the propagation time is predicted to be less than or equal to a preset threshold time; and a dashboard that receives the metadata, the bottleneck propagation map, and the evaluation results for the high-density area to perform visualization and alerting. Claim 2 In claim 1, the edge computing module comprises a sensor fusion module that fuses data acquired from the vision sensor, the LiDAR sensor, and the attitude sensor to produce image-spatial fusion data, and a non-identifying object tracking module that assigns an identifier to an object without personal identification information and tracks the movement of the object to generate the metadata, thereby forming an AI-based drone vision-LiDAR fusion crowd behavior pattern recognition and risk prediction system. Claim 3 delete Claim 4 delete

Images