Bird Strike Prevention System, Apparatus, and Method for Aircraft with Multi-Sensor Fusion and LLM-Based Autonomous Evasion / Repulsion Software

Abstract

A bird collision prevention system according to one embodiment of the present disclosure comprises: a sensor unit for collecting bird data—the sensor unit includes LiDAR, a thermal imaging camera, a Global Positioning System (GPS), an Inertial Navigation System (INS), an Automatic Dependent Surveillance-Broadcast (ADS-B) receiver, or a weather radar—; a preprocessing module for preprocessing the bird data collected from the sensor unit; an object recognition module for recognizing a bird based on the preprocessed bird data; a trajectory prediction module for predicting the movement pattern of the recognized bird; an LLM judgment module for generating a response scenario based on the movement pattern; a path planning module for calculating a danger zone where the bird may encroach and a possible avoidance path for an aircraft based on the response scenario; and an output unit for displaying the danger zone and the possible avoidance path on a display.

Description

Technology Field

[0001] The present disclosure relates to the field of aviation safety technology, and in particular to a technology that analyzes data collected from multiple sensors (LiDAR, camera, GPS, INS, ADS-B, weather radar, etc.) using CNN / LSTM-based AI and LLM (Large-scale Language Model) to provide real-time warnings and avoidance paths through the aircraft HUD / HEOD, and automatically activates a bird repellent device when necessary. Background Technology

[0002] In the field of aviation safety technology, data collection and analysis techniques utilizing multiple sensors are steadily advancing. Active research is underway to enhance aircraft safety based on data collected from various sensors, including LiDAR, cameras, GPS, INS, ADS-B, and weather radar. In particular, technology that analyzes complex data using CNN / LSTM-based artificial intelligence (AI) and Large-Scale Language Models (LLM), and provides this information to pilots in real-time via aircraft Head-Up Displays (HUDs) or Helmet-Mounted Displays (HEODs), is garnering significant attention. These technologies play a crucial role in supporting the safe operation of aircraft.

[0003] However, existing bird deterrent technologies still have limitations. Conventional bird deterrents utilizing acoustics or lasers mostly rely on manual operation or consist of single-sensor automated systems, resulting in limitations in terms of accuracy and response speed. In particular, in situations where pilots must make instantaneous decisions and take action, the likelihood of reaction delays increases, which can raise the risk of collisions with birds. This problem becomes even more pronounced in unexpected situations that may occur during flight operations and acts as a factor that increases the burden on pilots.

[0004] Furthermore, despite recent advancements in AI technology, there are few commercially available integrated systems that fuse multi-sensor data in real time, provide feedback to pilots via Augmented Reality (AR) or Head-Up Displays, and simultaneously execute bird deterrent commands automatically. This appears to be due to technical complexity and implementation difficulties, and significant challenges remain in effectively implementing such integrated systems at the current level of technology. Consequently, the development of more sophisticated and reliable technologies for aviation safety is required. The problem to be solved

[0005] The objective of the present disclosure is to provide an aircraft bird collision prevention system, device, and method combined with multi-sensor fusion and LLM-based autonomous avoidance and deterrence software, which can predict collisions by high-precision prediction of bird types, speeds, and movement paths through multi-sensor fusion and AI-based analysis, immediately display analysis results in the form of natural language and AR graphics on HUDs and HEODs, efficiently perform avoidance and deterrence, and minimize system delay through parallel data processing even when the number of sensors increases. means of solving the problem

[0006] In one embodiment of the present disclosure, a multi-sensor fusion and LLM-based bird collision prevention system may be provided. The system may include a sensor unit for collecting bird data, a preprocessing module for preprocessing the bird data collected from the sensor unit, an object recognition module for recognizing a bird based on the preprocessed bird data, a trajectory prediction module for predicting the movement pattern of the recognized bird, an LLM judgment module for generating a response scenario based on the movement pattern, a path planning module for calculating a danger zone that the bird may intrude upon and a possible avoidance path for an aircraft based on the response scenario, and an output unit for displaying the danger zone and the possible avoidance path on a display.

[0007] In one embodiment of the present disclosure, the object recognition module can recognize the bird by analyzing the preprocessed bird data with an artificial intelligence model based on a Convolutional Neural Network (CNN).

[0008] In one embodiment of the present disclosure, the trajectory prediction module can predict the movement pattern of the bird by analyzing the past movement data of the bird recognized by the object recognition module using an artificial intelligence model based on LSTM (Long Short-Term Memory).

[0009] In one embodiment of the present disclosure, the system may further include a corresponding scenario generation module that automatically generates the corresponding scenario using a Large Language Model (LM) based on the movement pattern.

[0010] In one embodiment of the present disclosure, the preprocessing module may include a timestamp synchronization algorithm for correcting the difference in collection times of the bird data collected from a plurality of sensors.

[0011] In one embodiment of the present disclosure, the preprocessing module can perform Region of Interest (ROI) cropping and resolution adjustment on the bird data.

[0012] In one embodiment of the present disclosure, the system may include a sensor fusion-based environment map generation module that generates an integrated environment map based on the bird data.

[0013] In one embodiment of the present disclosure, the path planning module may include a path calculation algorithm that calculates the avoidable path.

[0014] In one embodiment of the present disclosure, the path planning module may include a danger zone masking algorithm that indicates the danger zone.

[0015] In one embodiment of the present disclosure, the display may include a Head-Up Display (HUD) or a helmet-mounted display device.

[0016] In one embodiment of the present disclosure, the output unit may further include a function for outputting an automatic avoidance command based on the avoidable path.

[0017] In one embodiment of the present disclosure, the system may further include an automatic bird repellent control module that automatically controls a bird repellent device.

[0018] In one embodiment of the present disclosure, the system includes a plurality of sensor units, and the preprocessing module, object recognition module, trajectory prediction module, LLM judgment module and path planning module can process a plurality of bird data collected by the plurality of sensor units in parallel.

[0019] In one embodiment of the present disclosure, the preprocessing module, object recognition module, trajectory prediction module, LLM judgment module, and path planning module may include an asynchronous pipeline structure to increase the processing speed of bird data.

[0020] In one embodiment of the present disclosure, the object recognition module, the trajectory prediction module, and the LLM judgment module may include an INT8 quantization-based model lightweighting method to improve the computational efficiency of the artificial intelligence model.

[0021] In one embodiment of the present disclosure, the preprocessing module may include a structure that accelerates the preprocessing of bird data using a Field Programmable Gate Array (FPGA).

[0022] In one embodiment of the present disclosure, the path planning module may include a Fast Fail Safe avoidance algorithm that rapidly determines an avoidance path when an emergency situation occurs.

[0023] In one embodiment of the present disclosure, the LLM judgment module can calculate a reliability score for each corresponding scenario and determine whether to perform an avoidance by the aircraft based on the reliability score.

[0024] In one embodiment of the present disclosure, the system may include a function to perform a complex response mode including a bird deterrent command when it is determined that avoidance by the aircraft is impossible.

[0025] In one embodiment of the present disclosure, the object recognition module, the trajectory prediction module, or the LLM judgment module may include a self-reinforcement learning method that stores artificial intelligence learning log data and reuses the stored learning log data to improve the performance of the system.

[0026] In one embodiment of the present disclosure, a multi-sensor fusion and LLM-based bird collision prevention system may be provided. The system may use multiple weather response data sets configured such that a preprocessing module or an object recognition module enables the system to operate normally under multiple weather conditions.

[0027] In one embodiment of the present disclosure, the sensor unit may include a function to assist in collision avoidance between birds and aircraft by providing bird data collected by an ADS-B receiver to a path planning module.

[0028] In one embodiment of the present disclosure, a multi-sensor fusion and LLM-based bird collision avoidance system can be implemented with a modular hardware and software structure applicable to urban air mobility (UAM).

[0029] In one embodiment of the present disclosure, a multi-sensor fusion and LLM-based bird collision avoidance system may include a lightweight computational module and a power control structure so that it can be mounted on a drone.

[0030] In one embodiment of the present disclosure, the avoidable path and danger zone displayed on the display may be displayed in the form of Augmented Reality graphics, and a display standard may be applied.

[0031] In one embodiment of the present disclosure, the sensor unit may include a function for predicting the risk of collision with birds using bird data collected by a weather radar.

[0032] In one embodiment of the present disclosure, a multi-sensor fusion and LLM-based bird collision avoidance system may include a power management method that dynamically controls the driving power of a sensor unit, a preprocessing module, an object recognition module, a trajectory prediction module, an LLM judgment module, and a path planning module to optimize the energy efficiency of an aircraft.

[0033] In one embodiment of the present disclosure, the sensor unit includes a redundancy configuration for the sensor unit, so that the system can operate normally even if some of the sensors in the sensor unit malfunction.

[0034] In one embodiment of the present disclosure, a multi-sensor fusion and LLM-based bird collision avoidance system may be able to switch to manual control in emergency situations.

[0035] In one embodiment of the present disclosure, the LLM judgment module may include a function to generate a warning message in the form of natural language and display it through an output unit. Effects of the invention

[0036] According to one embodiment of the present disclosure, the risk of collision between birds and aircraft can be effectively reduced by predicting the type, speed, and movement path of birds with high precision through multi-sensor fusion analysis and generating response scenarios through an LLM-based decision module. According to one embodiment of the present disclosure, rapid and accurate response is possible without relying on the pilot's instantaneous judgment by utilizing CNN + LSTM-based AI to analyze time-axis patterns and calculate avoidance probabilities. Furthermore, according to one embodiment of the present disclosure, bird control and aircraft safety can be ensured in real time by displaying information on a HUD / HEOD in the form of AR graphics and natural language and executing commands to launch an automatic bird repellent device.

[0037] According to one embodiment of the present disclosure, detection, judgment, and avoidance of bird collision risks are automatically performed through multi-sensor fusion and a stepwise analysis and output structure, which can significantly reduce reaction delay and human error.

[0038] According to one embodiment of the present disclosure, CNN-based object recognition can increase the classification accuracy of the shape, distance, and speed characteristics of birds, thereby reducing false alarms and undetected rates.

[0039] According to one embodiment of the present disclosure, preemptive avoidance decisions are made by identifying the movement patterns of birds early through LSTM-based trajectory prediction.

[0040] According to one embodiment of the present disclosure, an LLM can provide consistent judgment and explainable decision-making even in a complex variable environment by automatically generating situational response scenarios.

[0041] According to one embodiment of the present disclosure, data fusion precision can be improved by eliminating errors caused by differences in collection times between sensors through timestamp synchronization.

[0042] According to one embodiment of the present disclosure, real-time processing is enabled by reducing computational load through ROI cropping and resolution adjustment and improving the target-centered signal-to-noise ratio (SNR).

[0043] According to one embodiment of the present disclosure, spatial recognition integrating bird, aircraft, and weather information can be implemented through a sensor fusion-based environment map to enhance situational awareness capabilities.

[0044] According to one embodiment of the present disclosure, a safety margin for the aircraft can be secured by rapidly calculating an avoidable path using a path calculation algorithm.

[0045] According to one embodiment of the present disclosure, the recurrence of path collisions can be prevented by intuitively specifying the space to be avoided through a danger zone masking algorithm.

[0046] According to one embodiment of the present disclosure, by using a HUD or a helmet-mounted display, information can be checked without shifting the gaze, thereby reducing the pilot's cognitive load and reaction time.

[0047] According to one embodiment of the present disclosure, survivability in emergency situations can be improved by performing immediate evasion actions without pilot intervention through the output of an automatic evasion command.

[0048] According to one embodiment of the present disclosure, active risk reduction in proximity threat situations is possible through an automatic repellent control module.

[0049] According to one embodiment of the present disclosure, by processing a plurality of sensor data in parallel, processing delays can be suppressed even when there is a data overload or an increase in the number of sensors, thereby ensuring system scalability.

[0050] According to one embodiment of the present disclosure, data collection and analysis are performed in overlap through an asynchronous pipeline structure, thereby reducing the overall processing latency.

[0051] According to one embodiment of the present disclosure, real-time inference is possible even in edge environments by reducing memory usage and computational load through INT8 quantization-based model lightweighting.

[0052] According to one embodiment of the present disclosure, the load of the CPU and GPU can be distributed by accelerating preprocessing operations using an FPGA, thereby improving the overall system response speed.

[0053] According to one embodiment of the present disclosure, by applying a Fast Fail-Safe algorithm, rapid and safe avoidance is possible even in the event of sensor malfunction or an imminent emergency situation.

[0054] According to one embodiment of the present disclosure, an LLM judgment module suppresses false automatic avoidance triggers and enables stable decision-making by calculating a reliability score for each scenario.

[0055] According to one embodiment of the present disclosure, when avoidance is impossible, the system switches to a combined response mode that includes a countermeasure command to minimize the final risk.

[0056] According to one embodiment of the present disclosure, the judgment performance of a system is continuously improved over time through self-reinforcement learning that recycles learning logs.

[0057] According to one embodiment of the present disclosure, robust recognition and prediction performance can be maintained even in adverse weather environments by using a dataset corresponding to multiple weather conditions.

[0058] According to one embodiment of the present disclosure, conservative and safe route planning is possible by considering airspace information between aircraft together through ADS-B data linkage.

[0059] According to one embodiment of the present disclosure, by adopting a modular hardware and software structure, it offers excellent portability to various platforms, such as Urban Air Mobility (UAM).

[0060] According to one embodiment of the present disclosure, stable real-time operation is possible even in a drone-mounted environment through a lightweight computation and power control structure.

[0061] According to one embodiment of the present disclosure, danger areas and avoidance paths can be intuitively recognized through display specifications that apply augmented reality (AR) graphics.

[0062] According to one embodiment of the present disclosure, early warning for a wide area is possible through bird risk prediction using weather radar.

[0063] According to one embodiment of the present disclosure, cruising performance can be improved by optimizing power consumption per module according to the situation through a power management method.

[0064] According to one embodiment of the present disclosure, system reliability can be improved by duplicating the sensor unit to ensure functional continuity even in the event of some sensor malfunction.

[0065] According to one embodiment of the present disclosure, it is possible to switch to manual control in emergency situations to ensure final safety and enhance operational reliability.

[0066] According to one embodiment of the present disclosure, an LLM can reduce response errors by generating a warning message in the form of natural language and providing an intuitive and easy-to-understand descriptive warning to the pilot.

[0067] According to one embodiment of the present disclosure, the accuracy of bird detection can be improved and false alarms reduced by fusing multiple sensor data and applying a time-axis-based prediction algorithm.

[0068] According to one embodiment of the present disclosure, the system can be installed on existing aircraft through a sensor package and software update, thus offering excellent commercialization and scalability of application.

[0069] According to one embodiment of the present disclosure, the system can be applied not only to urban air mobility (UAM) aircraft but also to various aviation platforms such as drones and helicopters, thus having a wide range of applications.

[0070] According to one embodiment of the present disclosure, by fusing CNN, LSTM, and LLM to combine image analysis, pattern prediction, and natural language-based instruction generation, it is possible to simultaneously achieve accuracy in judging bird collision risks and intelligence in responding to situations.

[0071] According to one embodiment of the present disclosure, unlike existing systems that present only a single avoidance path by simultaneously calculating the Happy Path and the infection area (danger zone), the pilot's situational awareness (SA) can be improved by visualizing the avoidance path and the danger zone together.

[0072] According to one embodiment of the present disclosure, even if the number of sensors increases, a parallel structure is applied so that the system operates without delay, thereby maintaining stable performance even in a large-scale multi-sensor environment.

[0073] According to one embodiment of the present disclosure, through integration with a Head-Up Display (HUD) or Helmet-Mounted Eye Optical Display (HEOD), a pilot can intuitively recognize and judge dangerous situations without shifting their gaze while driving.

[0074] According to one embodiment of the present disclosure, by combining and executing avoidance commands and repellent commands, an optimal response according to the bird approach stage can be automatically performed.

[0075] The significant effects obtainable from the present disclosure are compared with the prior art as follows.

[0076] division Conventional technology The present invention Processing structure Detection, warning, avoidance, and eradication each operate independently. Multi-sensor fusion → CNN recognition → LSTM prediction → LLM scenario generation → HUD / HEOD display → Auto-execution Recognition technology Simple pattern matching, threshold-based detection Simultaneous analysis of shape, distance, and speed using CNN-based object recognition Prediction technology No or simple linear prediction LSTM-based high-precision prediction of short-term trajectories (1–5 seconds) Decision-making Pilot manual judgment LLM generates and proposes real-time complex strategies (avoidance + countermeasures). interface Simple warning sound, instrument panel display Real-time display of route, hazards, and response reasons via HUD / HEOD AR Sensor configuration Single sensor or minority sensor LiDAR, IR, Radar, GPS, INS, ADS-B data fusion Delay time Delay of hundreds of ms to aquatic seconds Processing under 150ms with pipeline parallel processing and FPGA acceleration Methods for responding to failure Relying on the pilot after the warning Fast Fail-safe: Immediate elimination in case of evasion failure Whether learning and performance improvement are possible Fixed logic, lack of follow-up learning Continuous performance improvement with log-based self-reinforcement Applicable Platforms Limited to specific aircraft or environments SW + external sensor combined type, universally applicable to UAM, drones, and passenger aircraft

[0078] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below. Brief explanation of the drawing

[0079] FIG. 1 shows an overall configuration diagram of an aircraft bird collision prevention system combined with multi-sensor fusion and LLM-based autonomous avoidance and deterrence software according to one embodiment of the present invention. FIG. 2 shows a flowchart of bird data collected by a sensor unit and processed according to an embodiment of the present invention. FIG. 3 shows an example in which bird data analyzed by an aircraft bird collision prevention system combining multi-sensor fusion and LLM-based autonomous avoidance and deterrence software according to one embodiment of the present invention is output to a HUD or HEOD. FIG. 4 illustrates an example of an aircraft avoiding a collision with a bird or repelling a bird using an aircraft bird collision prevention system that combines multi-sensor fusion and LLM-based autonomous avoidance and repellent software according to an embodiment of the present invention. Specific details for implementing the invention

[0080] The embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order).

[0081] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components.

[0082] A multi-sensor fusion and LLM-based bird collision avoidance system refers to a system that prevents collisions between birds and aircraft by fusing various sensors. This system can collect and analyze data by utilizing various sensors such as LiDAR, thermal imaging cameras, GPS, INS, ADS-B receivers, and weather radar. The system can preprocess the collected data, recognize objects, and predict trajectories to determine the likelihood of a collision with birds. Furthermore, it can generate response scenarios through an LLM judgment module, calculate avoidance paths through a path planning module, and display information through an output unit.

[0083] The sensor unit may refer to a module containing various sensors to collect data. The sensor unit may consist of a lidar, thermal imaging camera, GPS, INS, ADS-B receiver, weather radar, etc. The sensor unit can collect bird data and transmit it to a preprocessing module. In addition, it can predict the probability of collision between birds and aircraft or provide data to a path planning module through an ADS-B receiver or weather radar.

[0084] LIDAR (Light Detection And Ranging) refers to a sensor that uses lasers to measure the distance and position of an object. LIDAR can primarily be used to collect 3D spatial data. LIDAR can be used as part of a sensor unit to accurately determine the location and movement path of birds. This can contribute to analyzing the possibility of collision between birds and aircraft.

[0085] A thermal imaging camera refers to a device that detects infrared signals to visualize temperature distribution. Thermal imaging cameras are primarily used to detect objects in nighttime or adverse weather environments. As part of its sensor, a thermal imaging camera can collect data by detecting the thermal signals of birds. This can be utilized to locate birds and analyze the likelihood of collisions.

[0086] GPS can refer to a system that provides location information using satellite signals. GPS can be used to track the position and speed of moving objects in real time. As part of the sensor unit, GPS can contribute to analyzing the probability of collision by collecting position data of aircraft and birds. Through this, it can provide data to the path planning module.

[0087] An INS (Inertial Navigation System) refers to a system that measures the position, velocity, and attitude of a moving object using inertial sensors. An INS can be used in conjunction with GPS to provide more accurate location information. As part of its sensor unit, the INS can collect movement data of birds and aircraft and provide it to a trajectory prediction module. This can contribute to predicting the movement patterns of birds.

[0088] An ADS-B (Automatic Dependent Surveillance-Broadcast) receiver refers to a device that transmits and receives information regarding an aircraft's position, speed, and altitude. ADS-B receivers can be used in air traffic control and collision avoidance systems. As part of the sensor unit, the ADS-B receiver can collect position data of birds and aircraft and provide it to the path planning module. This contributes to analyzing collision probabilities and calculating avoidance routes.

[0089] Weather radar can refer to a device that detects and analyzes weather conditions. Weather radar can provide information such as precipitation, wind, and clouds. As part of the sensor unit, weather radar can be used to predict bird migration paths and the likelihood of collisions. Through this, it can provide data to the path planning module, thereby contributing to collision prevention.

[0090] A preprocessing module refers to a module that performs the function of cleaning and transforming collected data before analysis. The preprocessing module can be used to improve data quality and increase the accuracy of analysis. The preprocessing module can synchronize bird data collected from the sensor unit and perform ROI cropping and resolution adjustment. In addition, it can accelerate preprocessing operations using an FPGA. The preprocessing module can filter noise from the bird data.

[0091] An object recognition module can refer to a module that performs the function of identifying specific objects by analyzing collected data. Object recognition modules can recognize objects primarily by utilizing artificial intelligence technology. Object recognition modules can recognize birds by analyzing preprocessed bird data using a CNN-based AI model. Additionally, computational efficiency can be improved through a model lightweighting method based on INT8 quantization.

[0092] A trajectory prediction module can refer to a module that performs the function of predicting the movement path of an object. A trajectory prediction module can predict future movement patterns primarily by analyzing past data. A trajectory prediction module can predict the movement patterns of birds using an LSTM-based artificial intelligence model. Through this, it can contribute to collision prevention by providing data to a path planning module.

[0093] The LLM decision module can refer to a module that performs the function of analyzing data and making decisions by utilizing large-scale language models. The LLM decision module can solve complex problems primarily by employing natural language processing techniques. The LLM decision module can generate response scenarios based on bird movement patterns. Additionally, it can determine whether an aircraft should perform an evasion maneuver by calculating a confidence score.

[0094] A path planning module may refer to a module that performs the function of designing and optimizing the movement path of an object. The path planning module can calculate an efficient path primarily by utilizing algorithms. The path planning module may include a masking algorithm that marks hazardous areas and an algorithm that calculates avoidable paths. Additionally, in emergency situations, it can rapidly determine a path through a high-speed fail-safe avoidance algorithm.

[0095] The output unit may refer to a module that performs the function of visually displaying analyzed data (including AR graphics or text) or outputting commands. The output unit can primarily provide information through a display device. The output unit can display avoidable routes and danger zones via a Head-Up Display (HUD) or Helmet-On Display (HEOD). Additionally, it can output automatic avoidance commands or generate and display warning messages in natural language. Furthermore, the output unit may refer to cockpit warning lights or the aircraft's acoustic and laser countermeasures systems.

[0096] A CNN-based artificial intelligence model can refer to an AI model based on a Convolutional Neural Network. CNN-based AI models can be used to analyze patterns in image or video data and to recognize objects. They can also be utilized to recognize birds by analyzing preprocessed bird data. Furthermore, CNN-based AI models can be used in object recognition modules of multi-sensor fusion and LLM-based bird collision avoidance systems.

[0097] An LSTM-based artificial intelligence model can refer to an artificial intelligence model based on Long Short-Term Memory. LSTM-based artificial intelligence models can be used to analyze and predict patterns in time-series data. LSTM-based artificial intelligence models can be utilized to predict movement patterns by analyzing past bird movement data. LSTM-based artificial intelligence models can be used in the trajectory prediction module of multi-sensor fusion and LLM-based bird collision avoidance systems.

[0098] The response scenario generation module may refer to a system module that automatically generates response measures for specific situations. The response scenario generation module can propose optimal response strategies based on various input data. The response scenario generation module can automatically generate response scenarios based on bird movement patterns using LLM. The response scenario generation module can be included as one of the core components of a multi-sensor fusion and LLM-based bird collision prevention system.

[0099] A timestamp synchronization algorithm can refer to an algorithm that synchronizes the time information of multiple data. A timestamp synchronization algorithm can be used to correct time differences in data collected from multiple sensors. It can be utilized to improve data accuracy by correcting differences in the collection times of bird data. A timestamp synchronization algorithm can be included in the preprocessing module of a multi-sensor fusion and LLM-based bird collision avoidance system.

[0100] ROI cropping refers to the process of selectively cropping a Region of Interest. It can be used to remove unnecessary parts of data and improve analysis efficiency. Specifically, ROI cropping can be utilized to enhance analysis efficiency by selectively processing specific regions of interest in bird data. ROI cropping can be performed in the preprocessing module of multi-sensor fusion and LLM-based bird collision avoidance systems.

[0101] Resolution adjustment refers to the process of optimizing processing speed or storage space by changing the resolution of data. It can be used to improve analysis efficiency by reducing data size or adjusting quality. Resolution adjustment can be utilized to enhance the processing speed of a system by adjusting the resolution of bird data. Resolution adjustment can be performed in the preprocessing module of a multi-sensor fusion and LLM-based bird collision avoidance system.

[0102] A sensor fusion-based environment map generation module refers to a module that generates an environment map by fusing data from multiple sensors. This module can provide real-time environmental information by integrating data collected from various sensors. It can be utilized for collision prevention by generating an integrated environment map based on bird data. The sensor fusion-based environment map generation module can be included in a multi-sensor fusion and LLM-based bird collision prevention system.

[0103] A path calculation algorithm can refer to an algorithm that calculates an optimal movement path. A path calculation algorithm can propose an efficient and safe path by considering various conditions. It can be utilized to calculate a path that avoids collisions with birds. The path calculation algorithm can be included in the path planning module of a multi-sensor fusion and LLM-based bird collision prevention system.

[0104] A danger zone masking algorithm can refer to an algorithm that identifies and marks specific danger zones. A danger zone masking algorithm can provide warnings by visually highlighting danger factors. A danger zone masking algorithm can be utilized to mark danger zones that birds are likely to intrude upon. A danger zone masking algorithm can be included in the path planning module of a multi-sensor fusion and LLM-based bird collision avoidance system.

[0105] A display can refer to a device that visually presents information. Displays can provide users with real-time data and support decision-making. Displays can be utilized in bird strike prevention systems to indicate avoidable paths and danger zones. Displays can be implemented in the form of a HUD or a helmet-mounted display.

[0106] A HUD refers to a display designed to allow users to view information in aircraft, automobiles, or other equipment without shifting their gaze. HUDs primarily project information onto a transparent screen, enabling users to simultaneously view the external environment. HUDs can be used to display avoidance routes and danger zones for aircraft. They can also be utilized as output units in multi-sensor fusion and LLM-based bird strike avoidance systems.

[0107] Helmet-mounted displays refer to display devices attached to a helmet worn by a user. Helmet-mounted displays are primarily used in military, aviation, or special operations environments. In multi-sensor fusion and LLM-based bird strike avoidance systems, helmet-mounted displays can serve as output units indicating an aircraft's avoidance paths and danger zones. Helmet-mounted displays can function as system displays in conjunction with a HUD.

[0108] An automatic avoidance command refers to a function that issues a command for a system to automatically perform avoidance actions in dangerous situations. Automatic avoidance commands are primarily used in aircraft, drones, or autonomous vehicles. Automatic avoidance commands can be provided through the output of multi-sensor fusion and LLM-based bird collision avoidance systems. Automatic avoidance commands can automatically execute an aircraft's avoidance path to prevent collisions with birds.

[0109] A bird repellent device can refer to a device designed to drive birds away from a specific area. Bird repellent devices are primarily used in locations such as airports, agricultural facilities, or power plants. Bird repellent devices can be controlled by an automatic repellent control module in multi-sensor fusion and LLM-based bird collision avoidance systems. Bird repellent devices can operate as part of a combined response mode when avoidance by an aircraft is impossible.

[0110] The automatic bird repellent control module refers to a system module that automatically controls a bird repellent device. The automatic bird repellent control module can execute the optimal repellent method by analyzing the location and movement patterns of birds. The automatic bird repellent control module can play a role in controlling the bird repellent device in a multi-sensor fusion and LLM-based bird collision prevention system. The automatic bird repellent control module can operate automatically to reduce the risk of collision with birds.

[0111] Multiple sensor units may refer to a part of a system composed of multiple sensors. Multiple sensor units can collect various data to enhance the accuracy and reliability of the system. Multiple sensor units can collect and process bird data in parallel in multi-sensor fusion and LLM-based bird collision prevention systems. Through a redundant configuration, multiple sensor units can ensure that the system operates normally even if some sensors malfunction.

[0112] An asynchronous pipeline structure can refer to a system structure that performs tasks asynchronously to increase data processing speed. An asynchronous pipeline structure can reduce bottlenecks between tasks and improve efficiency. An asynchronous pipeline structure can be used to rapidly process bird data in multi-sensor fusion and LLM-based bird collision avoidance systems. An asynchronous pipeline structure can optimize the data processing speed of modules such as preprocessing modules, object recognition modules, and trajectory prediction modules.

[0113] INT8 quantization-based model lightweighting refers to a technique that converts data representations into 8-bit integers to increase the computational efficiency of artificial intelligence models. INT8 quantization can reduce the size of models and improve processing speed. INT8 quantization-based model lightweighting can be used to increase the computational efficiency of object recognition modules, trajectory prediction modules, and LLM decision modules in multi-sensor fusion and LLM-based bird collision avoidance systems. INT8 quantization can optimize the overall performance of the system.

[0114] An FPGA can refer to a programmable semiconductor device capable of hardware design. FPGAs can increase computational speed by processing specific tasks in parallel. FPGAs can be used to accelerate computations in preprocessing modules in multi-sensor fusion and LLM-based bird collision avoidance systems. FPGAs can efficiently perform preprocessing tasks for bird data.

[0115] A high-speed fail-safe avoidance algorithm can refer to an algorithm that rapidly determines an avoidance path in emergency situations. High-speed fail-safe avoidance algorithms are designed to ensure the stability and safety of the system. In multi-sensor fusion and LLM-based bird collision avoidance systems, high-speed fail-safe avoidance algorithms can be included in the path planning module. High-speed fail-safe avoidance algorithms can operate rapidly to prevent aircraft collisions in emergency situations.

[0116] A reliability score can refer to a numerical value representing the reliability of specific data or judgments. Reliability scores can be used as an important criterion in a system's decision-making process. In multi-sensor fusion and LLM-based bird strike avoidance systems, reliability scores can be used by the LLM judgment module to evaluate the reliability of response scenarios. Reliability scores can also be utilized to determine whether an aircraft should perform an avoidance maneuver.

[0117] A combined response mode can refer to an approach that combines various response methods to respond appropriately depending on the situation. Combined response modes can provide effective solutions in complex situations. In multi-sensor fusion and LLM-based bird strike avoidance systems, a combined response mode may operate by including bird deterrent commands when the aircraft cannot avoid the bird. Combined response modes can contribute to enhancing the system's flexibility and responsiveness.

[0118] Artificial intelligence training log data can refer to data generated during the training process of an artificial intelligence model. Artificial intelligence training log data can be utilized for analyzing and improving the performance of the model. Artificial intelligence training log data can be used to enhance the performance of object recognition modules, trajectory prediction modules, and LLM decision modules in multi-sensor fusion and LLM-based bird collision avoidance systems. Artificial intelligence training log data can serve as the basis for self-reinforcement learning methods.

[0119] Self-reinforcement learning refers to a learning method in which a system independently utilizes training data to improve performance. Through continuous learning and adaptation, self-reinforcement learning can enhance system efficiency. In multi-sensor fusion and LLM-based bird collision avoidance systems, self-reinforcement learning can improve system performance by reusing stored training log data. Self-reinforcement learning can support the long-term stability and performance optimization of the system.

[0120] A multi-weather responsive dataset refers to a dataset designed to enable a system to operate normally under various weather conditions. Multi-weather responsive datasets can contribute to enhancing the reliability and stability of the system. In multi-sensor fusion and LLM-based bird collision avoidance systems, multi-weather responsive datasets can support preprocessing modules or object recognition modules to operate accurately even under diverse weather conditions. Multi-weather responsive datasets can strengthen the environmental adaptability of the system.

[0121] Urban Air Mobility (UAM) can refer to a transportation system utilizing aircraft in urban areas. It primarily includes small aircraft such as drones and UAMs. UAM can be applied through modular hardware and software architectures in multi-sensor fusion and LLM-based bird strike avoidance systems. UAM can be a significant application field in the design and implementation of systems.

[0122] Modular hardware can refer to a hardware structure where each component of a system can be designed and assembled independently. Modular hardware facilitates system scalability and maintenance. Modular hardware can be designed to be applicable to urban air mobility, ranging from multi-sensor fusion and LLM-based bird strike avoidance systems. Modular hardware can contribute to enhancing system flexibility and adaptability.

[0123] Software architecture can refer to the way a system's software is designed and configured. Software architecture is a critical factor in determining the functionality and performance of a system. Software architecture can be implemented to suit urban air mobility in multi-sensor fusion and LLM-based bird strike avoidance systems. Software architecture can contribute to ensuring the efficiency and stability of the system.

[0124] A drone can refer to an unmanned aerial vehicle that is remotely controlled or operates autonomously. Drones are used in various industries and application fields. Drones can be mounted in multi-sensor fusion and LLM-based bird collision avoidance systems through lightweight computational modules and power control structures. Drones can contribute to expanding the mobility and application possibilities of systems.

[0125] A lightweight computational module refers to a module that reduces size and weight while maintaining the system's computational performance. Lightweight computational modules are primarily used in drones or small equipment. They can be designed to be mounted on drones in multi-sensor fusion and LLM-based bird collision avoidance systems. Lightweight computational modules can contribute to enhancing the system's mobility and energy efficiency.

[0126] A power control structure can refer to a structure that efficiently manages a system's power consumption. A power control structure can increase the system's energy efficiency and extend its operating time. In multi-sensor fusion and LLM-based bird collision avoidance systems, a power control structure can be mounted on a drone to optimize power consumption. A power control structure can contribute to improving the system's sustainability and performance.

[0127] Augmented reality graphics refer to a technology that overlays digital information onto a real-world environment. Augmented reality graphics can enhance the user experience and provide information intuitively. In multi-sensor fusion and LLM-based bird collision avoidance systems, augmented reality graphics can be used to display avoidable paths and danger zones on the screen. Augmented reality graphics can enhance the system's visual information delivery capabilities.

[0128] Display specifications can refer to standards that define the method and format for displaying information on a display. Display specifications determine the system's user interface and information delivery efficiency. In multi-sensor fusion and LLM-based bird collision avoidance systems, display specifications can be applied to indicate avoidable paths and danger zones in the form of augmented reality graphics. Display specifications can contribute to enhancing the system's user-friendliness.

[0129] Energy efficiency can refer to the efficiency of the work performed relative to the energy consumed by a system. Energy efficiency affects the sustainability and operating costs of a system. Energy efficiency can be optimized through power management methods in multi-sensor fusion and LLM-based bird collision avoidance systems. Energy efficiency can contribute to improving the overall performance and sustainability of the system.

[0130] Power management refers to a method for efficiently controlling a system's power consumption. Power management can increase the system's energy efficiency and extend its operating time. In multi-sensor fusion and LLM-based bird collision avoidance systems, power management can dynamically control the driving power of components such as the sensor unit, preprocessing module, and object recognition module. Power management can contribute to optimizing the system's energy consumption.

[0131] A redundant configuration refers to a design that includes two or more components performing the same function to enhance system reliability. A redundant configuration can serve as a backup in the event of a system failure. In multi-sensor fusion and LLM-based bird collision prevention systems, a redundant configuration can ensure that the system operates normally even if some sensors in the sensor unit malfunction. A redundant configuration can strengthen the stability and reliability of the system.

[0132] Manual control refers to a method in which a user directly controls the system when it is not operating automatically. Manual control can be used to ensure the safety of the system in emergency situations. In multi-sensor fusion and LLM-based bird collision avoidance systems, manual control may include the ability to switch from automatic to manual control in emergency situations. Manual control can contribute to enhancing the flexibility and safety of the system.

[0133] Natural language warning messages refer to warning messages written in natural language that is easy for users to understand. Natural language warning messages enhance the user-friendliness of the system and enable rapid response. In multi-sensor fusion and LLM-based bird collision prevention systems, natural language warning messages can be generated by the LLM judgment module and displayed through the output unit. Natural language warning messages can strengthen the information delivery capabilities of the system.

[0134] According to one embodiment of the present invention, the object recognition module (130) may be configured to convert multiple channels of LiDAR, thermal camera images, and weather radar data, which have been noise removed, ROI cropped, and resolution adjusted in the preprocessing module (120), into input tensors of a CNN-based artificial intelligence model, and to infer whether each pixel and point cloud is a bird (430), an object ID, a size, and a relative speed class, and to transmit only objects classified as birds to the trajectory prediction module (140). Here, "input tensor" may refer to input data normalized into a matrix or a multidimensional array form of multiple channel image and point cloud data, "point cloud" may refer to a set of three-dimensional point data collected by LiDAR, and "object ID" may refer to an identifier for identifying the same bird object on the time axis.

[0135] According to one embodiment of the present invention, a trajectory prediction module (140) collects a time series of the position and velocity of a bird tagged with the same object ID from an object recognition module (130), uses this as input to an LSTM-based artificial intelligence model to calculate a predicted position sequence for the next few seconds, and can output the predicted trajectory as structured data including information on potential collision zones and times for each bird. Here, a time series may refer to a sequence of position and velocity data recorded in chronological order.

[0136] According to one embodiment of the present invention, the LLM judgment module (150) can automatically generate multiple response scenarios, such as detour flight, altitude adjustment, speed adjustment, and bird control, in the form of natural language and structured parameters by inputting the trajectory prediction result, sensor fusion-based environment map, flight plan of the aircraft (410), and restricted airspace information into the LLM as prompts. Here, the structured parameters may refer to information related to control commands expressed in a structured data format, such as numerical or enumerated types.

[0137] According to one embodiment of the present invention, the preprocessing module (120) can minimize the time error of the sensor fusion process by converting the timestamp assigned to each sensor of the sensor unit (110) into a common reference time and reordering the data of the sensor with a large delay based on the same point in time by interpolating or buffering it. Here, interpolation may refer to an operation that estimates the value of a missing point in time from surrounding data, and buffering may refer to a process of temporarily storing data for a certain period of time and outputting it at the synchronization point in time.

[0138] According to one embodiment of the present invention, a preprocessing module (120) can reduce the amount of computation and improve the signal-to-noise ratio for birds by designating only a certain area in front of the aircraft's direction of travel as the ROI, cropping the lidar, thermal image, and image data into a common projection area, and downsampling to the minimum resolution required for trajectory prediction and object recognition. Here, downsampling may refer to a process of reducing the amount of data and computation by lowering the spatial and temporal resolution of the raw data.

[0139] According to one embodiment of the present invention, a sensor fusion-based environment map generation module can generate an environment map in the form of a three-dimensional grid or point cloud by integrating lidar elevation information, weather radar reflectance, ADS-B-based aircraft traffic information and a terrain database, and can provide the bird density, weather risk, and airspace usage information for each cell as input to a path planning module (160). Here, an environment map in the form of a point cloud may refer to environmental information expressed as a plurality of points in a three-dimensional space, and a cell may refer to a minimum unit spatial block defined in a grid-based environment map.

[0140] According to one embodiment of the present invention, the path planning module (160) defines a cost function that takes the current flight plan of the aircraft (410) as a constraint using an environment map and a predicted trajectory, and can calculate an avoidable path that satisfies the minimum safe distance condition from birds by applying an A* algorithm, an RRT* or MPC-based path calculation algorithm. Here, the cost function may refer to a mathematical function that evaluates the superiority of paths by quantifying the fuel consumption, time, and safety margin of candidate paths, and the A* algorithm may refer to a heuristic-based shortest path search technique, the RRT* may refer to a probabilistic search-based path generation algorithm, and the MPC may refer to a future prediction model-based control optimization technique.

[0141] According to one embodiment of the present invention, the path planning module (160) defines a spatial area along a predicted bird trajectory where the probability of collision is evaluated to be greater than a threshold as a danger zone, generates a 3D mask, and superimposes it onto an environment map, and then visualizes the danger zone as a translucent color or a hatching pattern in the AR graphics displayed on the HUD / HEOD so that the pilot can intuitively recognize the space to avoid. Here, the 3D mask may refer to 3D array data that indicates whether each cell of the environment map is dangerous in order to represent the danger zone.

[0142] According to one embodiment of the present invention, the display is implemented as a HUD or HEOD, and the output unit (180) projects the avoidable path and danger zone calculated by the path planning module (160) so as to align with the aircraft's actual field of view, thereby allowing the pilot to view the external terrain while simultaneously checking guidance in the form of AR.

[0143] According to one embodiment of the present invention, the path planning module (160) converts the calculated avoidable path into a control command format of a flight control computer to generate an automatic avoidance command, and if the pilot accepts it through an acceptance interface provided by the output unit (180), transmits roll, pitch, and throttle commands to an autonomous control channel so that the aircraft (410) can automatically perform avoidance maneuvers. Here, the flight control computer may refer to an aircraft-mounted computer that receives control inputs and automatic control commands to control flight control surfaces and engine thrust, and the autonomous control channel may refer to an electronic path through which automatic control commands are transmitted to actual control surface actuators.

[0144] According to one embodiment of the present invention, an automatic bird repellent control module (170) communicates with an acoustic, laser, or lighting bird repellent device mounted around an airport runway or on the nose of an aircraft, and when the LLM judgment module (150) determines that repellent is necessary, it selects and automatically drives a repellent pattern of a specific wavelength and output according to the distance and direction of the bird, thereby guiding the bird out of its flight path in parallel with evasive maneuvers. Here, the bird repellent device may include an acoustic speaker, a laser irradiation device, a high-intensity lighting device, etc.

[0145] According to one embodiment of the present invention, the system distributes a plurality of sensor units (110) to the nose, wings, tail, etc., and configures an artificial intelligence processing module into a parallel pipeline for each sensor group to independently process bird data collected from each field of view and then integrates it in the final stage, thereby enabling coverage of a wide airspace without delay even when the number of sensors increases. Here, a parallel pipeline may refer to a structure in which multiple processing stages are performed independently of each other simultaneously.

[0146] According to one embodiment of the present invention, the system defines the data collection, preprocessing, recognition, prediction, LLM decision, and path planning stages as asynchronous tasks and introduces a message queue-based pipeline structure so that a lower stage processes the previous frame before the completion of a higher stage, thereby reducing the total latency to the level of tens to hundreds of milliseconds. Here, the message queue may refer to an asynchronous buffer structure for sequentially delivering data produced by each processing stage.

[0147] According to one embodiment of the present invention, the object recognition module (130), the trajectory prediction module (140), and the LLM judgment module (150) perform quantized cognitive learning in the learning phase and store and calculate weights and activation values ​​in INT8 format in the inference phase, thereby reducing memory usage and computational load while maintaining accuracy. Here, quantized cognitive learning may refer to a learning technique that considers low-precision computation.

[0148] According to one embodiment of the present invention, the preprocessing module (120) can process high-speed stream data in real time without CPU / GPU intervention by performing timestamp synchronization, ROI cropping, resolution adjustment, and filtering at the hardware level using FPGA logic.

[0149] According to one embodiment of the present invention, when the path planning module (160) detects an emergency situation such as a sensor malfunction, prediction failure, or sudden change in direction of the current, it can immediately execute a high-speed fail-safe avoidance algorithm and apply a standard maneuvering pattern that guarantees a minimum avoidance angle to rapidly reduce the risk of collision.

[0150] According to one embodiment of the present invention, an LLM judgment module (150) configures an evaluation prompt for each response scenario that includes feature quantities such as the bird collision risk reduction rate, the increase in fuel consumption, and the pilot operation difficulty, and receives a reliability score from the LLM to determine whether to automatically avoid and the priority of the presented scenarios. Here, the evaluation prompt may refer to an LLM input sentence containing quantitative and qualitative information related to the scenario.

[0151] According to one embodiment of the present invention, the automatic repellent control module (170) switches to a repellent priority mode when it is determined that avoidance maneuvers are difficult, and guides birds out of the danger zone by adjusting the output of the repellent device in stages.

[0152] According to one embodiment of the present invention, artificial intelligence modules store bird detection and avoidance events that occur during flight in a log database and periodically relearn them to enhance system performance. Here, the artificial intelligence learning log database may refer to a storage facility that structures and stores input, output, and result data.

[0153] According to one embodiment of the present invention, a preprocessing module (120) or an object recognition module (130) uses multi-weather corresponding learning data including bird data captured under various weather conditions, and selects an optimal model or parameters according to the current weather conditions during operation to maintain stable recognition performance even in severe weather conditions. Here, a sub-model may refer to a partial model for each weather condition.

[0154] According to one embodiment of the present invention, the sensor unit (110) includes an ADS-B receiver and provides surrounding aircraft location, altitude, and direction information, thereby enabling the calculation of an avoidance path that considers not only birds but also the risk of aircraft collision.

[0155] According to one embodiment of the present invention, the system is designed with a modular configuration that is portable to various platforms, such as Urban Air Mobility (UAM), and can be selectively configured on each platform in the form of hardware and software blocks having standardized interfaces. Here, a modular block may refer to a replaceable and combinable component.

[0156] According to one embodiment of the present invention, when the system is mounted on a drone, a lightweight computing module based on an NPU or a small GPU is used, and the power consumption of the sensor and model is dynamically adjusted according to the flight mode, battery level, and bird risk level through a power control structure, thereby enabling real-time operation even with limited power. Here, the NPU may refer to a Neural Processing Unit, which is a deep learning accelerator.

[0157] According to one embodiment of the present invention, the output unit (180) defines AR display specifications for HUD / HEOD, displays avoidance paths as three-dimensional curves based on an aircraft reference coordinate system, and displays danger zones as translucent volumes or polygons, and standardizes visual elements according to pilot interface guidelines. Here, the aircraft reference coordinate system may refer to a three-dimensional coordinate system defined based on the aircraft center and roll, pitch, and yaw axes.

[0158] According to one embodiment of the present invention, the sensor unit (110) can detect a reflection pattern unique to a bird flock using reflectance and Doppler velocity information of a weather radar, estimate the presence and direction of movement of a distant flock of birds, calculate the risk index, and provide it to the path planning module (160). Here, reflectance may refer to the radar signal reflection intensity, and Doppler velocity may refer to the relative velocity based on the Doppler effect.

[0159] According to one embodiment of the present invention, the system ensures both energy efficiency and safety by dynamically controlling the power consumption of sensors, preprocessing, and artificial intelligence modules according to the bird risk level and flight mode through a power management structure.

[0160] According to one embodiment of the present invention, the sensor unit (110) configures the main sensor in a redundant manner, and the preprocessing module (120) monitors the sensor health status and automatically switches to a backup sensor when an abnormality occurs, thereby ensuring that the bird collision prevention function is not interrupted. Here, the redundant configuration may mean the backup placement of a device with the same function.

[0161] According to one embodiment of the present invention, the output unit (180) provides an interface that allows the pilot to immediately switch to manual mode, blocks automatic control commands upon switching, and displays only the recommended path of the automatic system on the HUD / HEOD so that the pilot secures final control authority. Here, GUI elements may refer to operable user interface components such as buttons, toggles, and sliders.

[0162] According to one embodiment of the present invention, an LLM judgment module (150) generates an explanation prompt containing key information such as the nearest distance to a bird, the time required for avoidance, and whether a repellent device is used, and generates a natural language warning message based thereon and displays it on a HUD / HEOD or warning panel, thereby enabling the pilot to quickly understand the situation and respond. Here, the explanation prompt may refer to an input sentence that includes a situation description and numerical information to induce the LLM to generate a warning sentence.

[0163] FIG. 1 shows the overall configuration diagram of an aircraft bird collision prevention system combined with multi-sensor fusion and LLM-based autonomous avoidance and deterrence software according to one embodiment of the present invention.

[0164] The sensor unit (110) collects bird data, and the sensor unit (110) includes a LiDAR, a thermal imaging camera, a GPS (Global Positioning System), an INS (Inertial Navigation System), an ADS-B (Automatic Dependent Surveillance-Broadcast) receiver, or a weather radar.

[0165] The preprocessing module (120) preprocesses the bird data collected from the sensor unit. The preprocessing module (120) includes a timestamp synchronization algorithm to correct the difference in collection times of the bird data collected from a plurality of sensors. The preprocessing module (120) performs Region of Interest (ROI) cropping and resolution adjustment on the bird data. The preprocessing module (120) includes a function to generate an integrated environment map based on the bird data.

[0166] The object recognition module (130) recognizes birds based on preprocessed bird data. The object recognition module (130) can recognize birds by analyzing the preprocessed bird data with an artificial intelligence model based on a CNN (Convolutional Neural Network).

[0167] The trajectory prediction module (140) predicts the movement pattern of the recognized bird. The trajectory prediction module (140) predicts the movement pattern of the bird by analyzing the past movement data of the bird recognized by the object recognition module using an artificial intelligence model based on LSTM (Long Short-Term Memory).

[0168] The LLM judgment module (150) generates response scenarios based on movement patterns. The LLM judgment module (150) calculates a reliability score for each response scenario and can determine whether to perform evasion by the aircraft based on the reliability score.

[0169] The path planning module (160) calculates a danger zone where birds are likely to intrude and a possible avoidance path for the aircraft based on a response scenario.

[0170] The automatic bird repellent control module (170) can automatically control the bird repellent device. If the automatic bird repellent control module (170) determines that avoidance by the aircraft is impossible, it can perform a complex response mode including a bird repellent command.

[0171] The output unit (180) can display danger zones and avoidable paths on a display. The display includes a Head-Up Display (HUD) or a Helmet-Mounted Display (HEOD). The output unit (180) can display avoidable paths and danger zones in the form of Augmented Reality (AR) graphics. The output unit (180) can display a warning message in the form of natural language generated by the LLM judgment module (150). The output unit (180) can output a GUI that switches the system to manual control in emergency situations.

[0172] FIG. 2 shows a flowchart of bird data processing according to one embodiment of the present invention.

[0173] Bird data collected by the sensor unit (110) is input into a preprocessing module (120) for preprocessing (210). The preprocessing module synchronizes / matches the bird data using a timestamp synchronization algorithm to correct the difference in the time of collection of the bird data (220), and then preprocesses the bird data (230). The preprocessed bird data is recognized by an object recognition module (130), a movement pattern is predicted by a trajectory prediction module (140), and a response scenario is generated by an LLM judgment module (150) (240). The path planning module (160) calculates a danger zone where birds may encroach and a possible avoidance path for the aircraft based on the response scenario (250). The automatic evasion control module (170) operates the automatic evasion device based on the calculated danger zone and the aircraft's avoidance path (260), and the output unit (180) displays the calculated danger zone and the aircraft's avoidance path on a display (HUD (Head-Up Display) or HEOD (Helmet-Mounted Display)).

[0174] FIG. 3 illustrates an example in which bird data analyzed by an aircraft bird collision prevention system combining multi-sensor fusion and LLM-based autonomous avoidance and deterrence software, according to an embodiment of the present invention, is output to a display (HUD or HEOD). The location (310) of a bird detected by the sensor unit (110) can be output to the display. The collision risk zone, the probability of collision within the risk zone (320), and the aircraft's avoidance path (320) calculated by the path planning module (160) can be output to the display. The risk zone can be displayed in red, and the avoidance path can be displayed in green. The analyzed bird data output to the display can be displayed in the form of an Augmented Reality (AR) graphic in the center of the display.

[0175] FIG. 4 illustrates an example of an aircraft avoiding a collision with a bird using a bird collision prevention system combined with multi-sensor fusion and LLM-based autonomous avoidance and deterrence software according to an embodiment of the present invention. During the operation of the aircraft (410), the sensor unit (110) of the bird collision prevention system according to the present invention collects data of a bird (430) using a lidar, a thermal imaging camera, etc. (420), and the collected bird data is preprocessed by a preprocessing module (120). An object recognition module recognizes the bird based on the preprocessed bird data, a trajectory prediction module predicts the movement pattern of the bird, an LLM judgment module (150) generates a response scenario based on the movement pattern of the bird, and a path planning module (160) outputs a collision risk zone, a possibility of collision within the risk zone, and a possible avoidance path for the aircraft to a display mounted by the pilot based on the response scenario. The pilot safely avoids the bird and operates the aircraft based on the displayed possible avoidance path.

[0176] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below.

Claims

Claim 1 In a multi-sensor fusion and LLM-based bird collision avoidance system, a sensor unit for collecting bird data—the sensor unit includes LiDAR, a thermal imaging camera, a Global Positioning System (GPS), an Inertial Navigation System (INS), an Automatic Dependent Surveillance-Broadcast (ADS-B) receiver, or a weather radar—; a preprocessing module for preprocessing the bird data collected from the sensor unit; an object recognition module for recognizing a bird based on the preprocessed bird data; a trajectory prediction module for predicting the movement pattern of the recognized bird; an LLM judgment module for generating a response scenario based on the movement pattern; and a path planning module for calculating a danger zone where the bird is likely to intrude and a possible avoidance path for an aircraft based on the response scenario. A multi-sensor fusion and LLM-based bird collision avoidance system comprising an output unit that displays the above-mentioned danger zone and avoidable path on a display, wherein the LLM judgment module configures an evaluation prompt including feature quantities including bird collision risk reduction rate, fuel consumption increase, and pilot operation difficulty for each response scenario and inputs it into the LLM, and receives a reliability score for each response scenario from the LLM to determine whether the aircraft performs automatic avoidance and the priority of the response scenario, wherein the path planning module is configured to convert the calculated avoidable path into a control command format of a flight control computer to generate an automatic avoidance command, and wherein the output unit is configured to provide an acceptance interface for receiving the pilot's acceptance of the automatic avoidance command, and wherein the system is configured such that when the pilot accepts the automatic avoidance command through the acceptance interface, roll, pitch, and throttle commands are transmitted to the aircraft's autonomous control channel so that the aircraft automatically performs avoidance maneuvers. Claim 2 A multi-sensor fusion and LLM-based bird collision prevention system according to claim 1, wherein the object recognition module recognizes the bird by analyzing the preprocessed bird data with a CNN (Convolutional Neural Network)-based artificial intelligence model. Claim 3 A multi-sensor fusion and LLM-based bird collision prevention system according to claim 2, wherein the trajectory prediction module predicts the movement pattern of the bird by analyzing the past movement data of the bird recognized by the object recognition module using an LSTM (Long Short-Term Memory)-based artificial intelligence model. Claim 4 In paragraph 3, the system further comprises a response scenario generation module that automatically generates the response scenario using a Large Language Model (LM) based on the movement pattern, a multi-sensor fusion and LLM-based bird collision prevention system. Claim 5 A multi-sensor fusion and LLM-based bird collision prevention system according to claim 1, wherein the preprocessing module includes a timestamp synchronization algorithm for correcting the difference in collection times of the bird data collected from a plurality of sensors. Claim 6 A multi-sensor fusion and LLM-based bird collision prevention system according to claim 1, wherein the preprocessing module includes performing Region of Interest (ROI) cropping and resolution adjustment on the bird data. Claim 7 A multi-sensor fusion and LLM-based bird collision prevention system according to claim 1, wherein the preprocessing module includes a function to generate an integrated environment map based on the bird data. Claim 8 A multi-sensor fusion and LLM-based bird collision avoidance system according to claim 1, wherein the path planning module includes a path calculation algorithm that calculates the avoidable path. Claim 9 A multi-sensor fusion and LLM-based bird collision prevention system according to claim 1, wherein the path planning module includes a danger zone masking algorithm that indicates the danger zone. Claim 10 In claim 1, the display comprises a Head-Up Display (HUD) or a helmet-mounted display device, a multi-sensor fusion and LLM-based bird collision prevention system. Claim 11 A multi-sensor fusion and LLM-based bird collision avoidance system according to claim 1, wherein the path planning module further includes a function for calculating an automatic avoidance command based on the avoidable path. Claim 12 A multi-sensor fusion and LLM-based bird collision prevention system according to claim 1, wherein the system further comprises an automatic bird repellent control module that automatically controls a bird repellent device. Claim 13 In claim 1, the system comprises a plurality of sensor units, and the preprocessing module, the object recognition module, the trajectory prediction module, the LLM judgment module, and the path planning module are capable of parallel processing of a plurality of bird data collected by the plurality of sensor units, a multi-sensor fusion and LLM-based bird collision prevention system. Claim 14 A multi-sensor fusion and LLM-based bird collision prevention system according to claim 1, wherein the preprocessing module, the object recognition module, the trajectory prediction module, the LLM judgment module, and the path planning module include an asynchronous pipeline structure to increase the processing speed of the bird data. Claim 15 A multi-sensor fusion and LLM-based bird collision prevention system according to claim 1, wherein the object recognition module, the trajectory prediction module, and the LLM judgment module include an INT8 quantization-based model lightweighting method to improve the computational efficiency of the artificial intelligence model. Claim 16 A multi-sensor fusion and LLM-based bird collision prevention system according to claim 1, characterized in that the preprocessing module includes a structure that accelerates the preprocessing operation of the bird data using a FieldProgrammable Gate Array (FPGA). Claim 17 A multi-sensor fusion and LLM-based bird collision prevention system according to claim 1, wherein the path planning module includes a Fast Fail Safe avoidance algorithm that rapidly determines an avoidance path when an emergency situation occurs. Claim 18 A multi-sensor fusion and LLM-based bird collision avoidance system according to claim 1, wherein the LLM judgment module calculates a reliability score for each of the corresponding scenarios and determines whether to perform avoidance by the aircraft based on the reliability score. Claim 19 A multi-sensor fusion and LLM-based bird collision prevention system according to claim 1, further comprising an automatic bird deterrent control module that performs a complex response mode including a bird deterrent command when it is determined that avoidance by the aircraft is impossible. Claim 20 A multi-sensor fusion and LLM-based bird collision prevention system according to claim 1, wherein the object recognition module, the trajectory prediction module, or the LLM judgment module includes a self-reinforcement learning method that stores artificial intelligence learning log data and recycles the stored learning log data to improve the performance of the system. Claim 21 A multi-sensor fusion and LLM-based bird collision prevention system according to claim 1, wherein the preprocessing module or the object recognition module uses a multiple weather response data set configured to enable the system to operate normally under multiple weather conditions. Claim 22 A multi-sensor fusion and LLM-based bird collision avoidance system according to claim 1, characterized in that the sensor unit includes a function of providing the bird data collected by the ADS-B receiver to the path planning module to assist in collision avoidance between the bird and the aircraft. Claim 23 A multi-sensor fusion and LLM-based bird collision avoidance system according to claim 1, characterized in that the system is implemented with a modular hardware and software structure applicable to urban air mobility (UAM). Claim 24 A multi-sensor fusion and LLM-based bird collision avoidance system according to claim 1, characterized in that the system includes a lightweight computation module and a power control structure so as to be mounted on a drone. Claim 25 A multi-sensor fusion and LLM-based bird collision prevention system according to claim 1, characterized in that the avoidable path and danger zone displayed on the display apply a display standard in the form of augmented reality (AR) graphics. Claim 26 A multi-sensor fusion and LLM-based bird collision prevention system according to claim 1, characterized in that the sensor unit includes a function that predicts the possibility of collision risk of the bird using the bird data collected by the weather radar. Claim 27 A multi-sensor fusion and LLM-based bird collision avoidance system according to claim 1, characterized in that the system includes a power management method that dynamically controls the driving power of the sensor unit, the preprocessing module, the object recognition module, the trajectory prediction module, the LLM judgment module, and the path planning module to optimize the energy efficiency of the aircraft. Claim 28 A multi-sensor fusion and LLM-based bird collision prevention system according to claim 1, wherein the sensor unit includes a redundancy configuration for the sensor unit, so that the system operates normally even if some of the sensors in the sensor unit malfunction. Claim 29 A multi-sensor fusion and LLM-based bird collision prevention system according to claim 1, characterized in that the system can be switched to manual control in emergency situations. Claim 30 A multi-sensor fusion and LLM-based bird collision prevention system according to claim 4, wherein the LLM judgment module includes a function to generate a warning message in the form of natural language and display it through the output unit.

Images