Visual navigation system and method based on an vlm with understanding of flight dynamics and geospatial information

Abstract

The present invention relates to a VLM-based visual navigation system and method for understanding flight dynamics and geospatial information. A visual navigation method according to the present invention comprises the steps of: a visual navigation device continuously collecting sensor data using a sensor mounted on a target aircraft and generating a sequential dataset based on the sensor data; the visual navigation device generating a prompt for requesting navigation information from a visual language model; and the visual navigation device extracting reference data from a reference database and inputting the prompt, the sequential dataset, and the reference data into the visual language model to generate first navigation information.

Description

Technology Field

[0001] The present invention belongs to the field of visual navigation technology and, in particular, relates to a visual navigation device and method utilizing a position estimation algorithm for an aircraft, drone, satellite, etc., and an algorithm for understanding and processing flight dynamics and geospatial information required for path planning.

[0002] This research was conducted as part of the DNA-based national intelligent core technology development project by the Electronics and Telecommunications Research Institute (ETRI). [23ZR1210, DNA-based national intelligent core technology development] Background Technology

[0003] Advanced Air Mobility (AAM) or Urban Air Mobility (UAM) refers to low-to-medium altitude aircraft flying at altitudes ranging from 300 to 600 meters, primarily operating autonomously in urban and suburban areas. As these aircraft achieve fully unmanned autonomous flight, situations may arise where they cannot rely solely on existing GPS-based precision satellite navigation systems. In particular, more precise and safer navigation is required in environments where GPS signals are blocked (GPS Denied) or performance is degraded (GPS Degraded). Consequently, the importance of Visual Navigation or Complementary Relative Navigation is emerging.

[0004] Currently, there is a method used in robotics and autonomous driving technology that processes image data through feature extraction using artificial intelligence and implements navigation based on it. This is a concept similar to how a real helicopter pilot performs Visual Flight Rules (VFR) at a low altitude, mimicking the way a pilot flies while visually checking the surrounding environment.

[0005] However, flight scenarios involve highly complex and diverse environments. For example, generalizable image-based navigation is required to maintain accurate navigation under various climatic conditions, such as changes in weather or lighting, fog, and rain. To date, AI models specialized for specific environments have been primarily used; however, this approach demonstrates high performance only in limited situations and struggles to comprehensively handle a wide range of scenarios.

[0006] To address this, methods utilizing foundation models trained on large-scale data are being proposed. In particular, by combining Large Language Models (LLMs) or Visual Language Models (VLMs), it has become possible to implement navigation systems that mimic the thought processes and visual perceptions of actual human pilots. Such models enable Generalizable Visual Navigation, which can deliver high performance in diverse environments without being limited to specific situations or scenarios. Prior art literature

[0007] US 2024 / 0095077 A1 (2024.3.21.)US 2024 / 0095460 A1 (2024.3.21.) The problem to be solved

[0008] The present invention aims to provide a visual language model (VLM)-based visual navigation system and method that significantly improves the operational stability of future air mobility (AAM) through image-based navigation that mimics the judgment and environmental perception capabilities of a human pilot, and provides stable and accurate navigation even in situations where GPS performance is degraded.

[0009] The objectives of the present invention are not limited to those mentioned above, and other unmentioned objectives will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0010] A visual navigation method according to an embodiment of the present invention comprises: a step in which a visual navigation device continuously collects sensor data using a sensor mounted on a target aircraft and generates a sequential dataset based on the sensor data; a step in which the visual navigation device generates a prompt for requesting navigation information to a visual language model; and a step in which the visual navigation device extracts reference data from a reference database and inputs the prompt, the sequential dataset, and the reference data into the visual language model to generate first navigation information.

[0011] In one embodiment of the present invention, the visual navigation method further comprises: a step in which the visual navigation device generates second navigation information using visual odometry based on the sequential dataset; and a step of inputting the first navigation information and the second navigation information into a delayed Kalman filter to generate third navigation information, which is navigation information of a target aircraft at a current time.

[0012] In one embodiment of the present invention, the visual navigation method further includes the step of the visual navigation device generating a three-dimensional map using a sub-keyframe generation technique based on the sequential dataset.

[0013] In one embodiment of the present invention, the step of generating the third navigation information further comprises the visual navigation device collecting inertial data of a target aircraft using the sensor and inputting the inertial data into the delayed Kalman filter.

[0014] In one embodiment of the present invention, the step of generating the prompt includes the visual navigation device including at least one of the state of the target aircraft, the name of a dynamic model describing the operation of the target aircraft, and the formula of the dynamic model in the prompt.

[0015] A visual navigation device according to one embodiment of the present invention includes a memory for storing computer-readable commands; and at least one processor implemented to execute said commands.

[0016] The above at least one processor is configured to continuously collect sensor data using a sensor mounted on a target aircraft by executing the above commands, generate a sequential dataset based on the sensor data, extract reference data from a reference database, generate a prompt including the reference data and the sequential dataset to request navigation information from a visual language model, and input the prompt into the visual language model to generate first navigation information.

[0017] In one embodiment of the present invention, the at least one processor is configured to generate second navigation information using visual odometry based on the sequential dataset, and to generate third navigation information, which is navigation information of a target aircraft at a current time, by inputting the first navigation information and the second navigation information into a delayed Kalman filter.

[0018] In one embodiment of the present invention, the at least one processor is configured to generate a three-dimensional map using a sub-keyframe generation technique based on the sequential dataset.

[0019] In one embodiment of the present invention, the at least one processor is configured to collect inertial data of a target aircraft using the sensor during the process of generating the third navigation information, and to input the inertial data into the delayed Kalman filter.

[0020] In one embodiment of the present invention, the at least one processor is configured to include at least one of the state of a target aircraft, the name of a dynamic model describing the operation of the target aircraft, and the formula of the dynamic model in the prompt during the process of generating the prompt.

[0021] In one embodiment of the present invention, the at least one processor is configured to generate the second navigation information using visual odometry based on the sequential dataset from the point in time until the point in time when the prompt, the sequential dataset, and the reference data are input into the visual language model, and an output is obtained from the visual language model.

[0022] A visual navigation system according to one embodiment of the present invention comprises: a first memory, a first processor, and a storage device, and a VLM server that drives a visual language model (VLM) in the storage device; and a visual navigation device.

[0023] The visual navigation device includes a second memory for storing computer-readable commands; and a second processor implemented to execute said commands.

[0024] The second processor is configured to execute the above commands to continuously collect sensor data using a sensor mounted on a target aircraft, generate a sequential dataset based on the sensor data, extract reference data from a reference database, transmit a prompt including the reference data and the sequential dataset to the VLM server via wireless communication, receive a response to the prompt from the VLM server, and generate first navigation information including position information and attitude information of the target aircraft based on the response. Effects of the invention

[0025] The present invention is expected to be utilized in various fields listed below, including the fields of drones and urban air mobility. The effects that can be obtained through the application of the present invention in each field are as follows.

[0026] 1. Drone and UAM (Urban Air Mobility) Industry Field: The universal visual navigation system of the present invention enables stable autonomous flight even in situations of climate change or unstable GPS signals, and thus can be utilized in commercial autonomous aircraft such as drone delivery and urban air mobility (UAM). In particular, by utilizing the present invention, reliable flight path planning and real-time location tracking are possible even in complex urban environments or emergency situations, thereby enabling the safe and efficient operation of aircraft.

[0027] 2. Defense and Security Sector: Autonomous flying vehicles capable of operating in various climate and light conditions can play a crucial role in missions such as military reconnaissance and security surveillance. The present invention can be effectively utilized even in such high-risk environments thanks to real-time performance enhancement and precise location estimation.

[0028] 3. Disaster Rescue and Emergency Services Field: The 3D reconstruction technology of the present invention enables accurate spatial recognition and path planning at the rescue site, thereby supporting a rapid response to the affected area in the event of a natural disaster. Through the present invention, complex terrain can be identified using drones, and emergency supplies or rescue personnel can be effectively deployed.

[0029] 4. Logistics and Delivery Field: If the technology of the present invention is applied to autonomous delivery drones or robot logistics systems, stable and efficient logistics services can be provided regardless of complex urban environments or weather changes. In particular, real-time path correction functions based on visual navigation and odometry can increase the reliability of autonomous logistics systems.

[0030] The effects obtainable from the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description below. Brief explanation of the drawing

[0031] FIG. 1 is a block diagram showing the configuration of a visual navigation system according to one embodiment of the present invention. FIG. 2 is a diagram showing a visual navigation device according to an embodiment of the present invention acquiring an aerial image. FIG. 3 is a conceptual diagram of a visual navigation method according to an embodiment of the present invention. FIG. 4 is a flowchart illustrating a visual navigation method according to an embodiment of the present invention. Fig. 5 is a diagram regarding the acquisition of aerial images. FIG. 6 is a diagram illustrating the process of estimating the position of an aircraft by inputting aerial images and reference data into a VLM. Figure 7 is an example diagram of a prompt (query) entered into the VLM and the VLM's response to it. FIG. 8 is a diagram illustrating reference data and aerial images input into the VLM and the output values ​​of the VLM for the same. Specific details for implementing the invention

[0032] The present invention belongs to the field of visual navigation technology and, in particular, relates to a visual navigation device and method utilizing an algorithm for understanding and processing flight dynamics and geospatial information required for position estimation and path planning of flying objects such as aircraft, drones, and satellites.

[0033] The present invention enables image-based navigation by utilizing a Large Language Model (LM) that mathematically understands the dynamics and spatial characteristics of an aircraft based on pixel data extracted from large-scale aerial or satellite images. Furthermore, since the LLM understands the geometric structure and relationship between the pixel information of the image and the actual Earth coordinate system, the position of the aircraft can be estimated in real time based on the Earth coordinate system.

[0034] The present invention can be applied to various autonomous systems such as autonomous flying vehicles, unmanned aerial vehicles (UAVs), autonomous vehicles, and satellite-based navigation systems, and can be utilized in academic and industrial technical fields such as artificial intelligence (AI), deep learning, computer vision, aerospace engineering, robotics, and geographic information systems (GIS).

[0035] The features of the present invention are summarized as follows.

[0036] 1. Generalizable Visual Navigation

[0037] Existing visual navigation systems have limitations in performance across various situations due to their reliance on algorithms specialized for specific environments. In particular, a more general-purpose approach is required to maintain seamless image navigation performance even under diverse climatic conditions, such as weather, changes in illumination, fog, and rain. This invention proposes a generalizable visual navigation system that utilizes Large Language Models (LLM) and Visual Language Models (VLM) to enhance visual environment perception capabilities, similar to those of a human pilot. The visual navigation system according to this invention enables stable and reliable navigation across various scenarios, without being limited to specific situations.

[0038] 2. Visual Odometry and Delayed Kalman Filter-Based Localization to Overcome the Real-Time Limitations of VLM

[0039] Generalizable Visual Navigation (GNA) systems based on Visual Language Models (VLMs) can be utilized in various situations, but real-time performance may degrade during the process of handling large-scale data. This limitation can cause problems in situations where autonomous aircraft require real-time decision-making. Therefore, the present invention includes using Visual Odometry to predict changes in relative position while camera images are being processed, and compensating for position estimation errors through these predicted values. In particular, by fusing the corrected position provided by the VLM with the information predicted by Visual Odometry using a Delayed Kalman Filter (Kalman Filter with Delayed Measurement) that utilizes the camera image timestamps, the current position (Odometry) of the aircraft can be estimated by taking delay time into account. Accordingly, the present invention can effectively overcome the problem of real-time performance degradation.

[0040] 3. Utilization of Sequential Data and Sub-keyframe Generation via NeRF and SLAM

[0041] Navigation of an aircraft must utilize image data acquired in a continuously changing environment, and it is advantageous to process a sequential set of images rather than relying on a single frame of an image. This invention proposes a method for generating and utilizing sub-keyframes, which are partially reconstructed maps, by applying Neural Radiance Fields (NeRF) and Simultaneous Localization and Mapping (SLAM) algorithms. This allows for more efficient analysis by selecting critical moments of the flight path and enables more accurate and reliable navigation through continuous image data. Furthermore, in this process, the relationship between the pixel information of the image and actual spatial geographic information is calculated to correspond the results of image navigation to the spatial geography of the Earth coordinate system, thereby enabling it to be utilized as an alternative to GPS sensors.

[0042] 4. Understanding Flight Dynamics

[0043] Recently, the performance of mathematical and dynamic reasoning in Large Language Models (LLMs) has been significantly improved. By utilizing the reasoning capabilities of LLMs, accuracy can be enhanced compared to performing complex or image navigation using only visual information. Furthermore, just as actual human perception and thought are simulated, inertial information can be inferred and utilized based on visual information. To simulate this, the present invention proposes a method for utilizing additional information regarding flight dynamics in navigation by acquiring the state information of an aircraft and accumulating and transmitting it to a language model. Additionally, since the inference results may vary depending on the level of the model for the aircraft, names or formulas for the precision model, simplified model, kinematic model, and dynamic model are transmitted to the language model when initializing it. By inputting the aircraft's state information into the language model, the language model can be configured to output the aircraft's navigation information.

[0044] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Meanwhile, the terms used in this specification are for describing the embodiments and are not intended to limit the present invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. The terms "comprises" and / or "comprising" as used in this specification do not exclude the presence or addition of one or more other components, steps, actions, and / or elements in addition to the mentioned components, steps, actions, and / or elements.

[0045] Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms may be used for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0046] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. Conversely, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions describing the relationship between components, such as "between" and "exactly between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0047] In describing the present invention, detailed descriptions of related prior art are omitted if it is determined that such descriptions may unnecessarily obscure the essence of the invention.

[0048] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In order to facilitate overall understanding in describing the present invention, the same reference numerals will be used for the same means regardless of the drawing number.

[0049] FIG. 1 is a block diagram showing the configuration of a visual navigation system according to one embodiment of the present invention.

[0050] Referring to FIG. 1, the visual navigation system (10) includes a visual navigation device (100) and a VLM server (50) that operates a visual language model (VLM). The visual navigation device (100) is mounted on a target aircraft and can transmit and receive data via wireless communication with the VLM server (50). The visual navigation device (100) transmits a prompt to be input into the visual language model and sensor data (e.g., image, point cloud) or a sequential dataset formed by sequentially merging said sensor data to the VLM server (50). The VLM server (50) inputs said prompt and said sensor data or said sequential dataset into the visual language model (VLM) to obtain a response from the visual language model (VLM) and can transmit said response (output) to the visual navigation device (100).

[0051] As another example, the visual navigation device (100) may store a visual language model (VLM) in a built-in storage device (140). In this case, the visual navigation device (100) may input the prompt and the sensor data or the sequential dataset into the visual language model (VLM) and obtain the response itself.

[0052] A visual navigation device (100) may include at least one processor (110), a communication device (120), a memory (130), a storage device (140), and a sensor (150) that communicate via a bus (160). The visual navigation device (100) illustrated in FIG. 1 is according to one embodiment, and the components of the visual navigation device (100) according to the present invention are not limited to the embodiment illustrated in FIG. 1 and may be added, changed, or deleted as needed.

[0053] The processor (110) may be a central processing unit (CPU) or a semiconductor device that executes computer-readable instructions stored in memory (130) or storage device (140). Memory (130) and storage device (140) may include various forms of volatile or non-volatile storage media. For example, memory (130) may include read-only memory (ROM) and random access memory (RAM). In the embodiments of this description, memory (130) may be located inside or outside the processor (110), and memory (130) may be connected to the processor (110) through various known means. Memory (130) is various forms of volatile or non-volatile storage media, and for example, memory (130) may include read-only memory (ROM) or random access memory (RAM).

[0054] Accordingly, embodiments of the present invention may be implemented as a method implemented on a computer or as a non-transient computer-readable medium in which computer-executable instructions are stored. In one embodiment, when executed by a processor (110), the computer-readable instructions may perform a method according to at least one aspect of the present description.

[0055] The communication device (120) can transmit or receive wireless signals. The communication device (120) can transmit and receive data via wireless communication with the communication device of the VLM server (50).

[0056] Additionally, the storage device (140) stores reference data through a reference database. Reference data is data that serves as a comparison target for sensor data acquired through the sensor (150). For example, the reference data may be aerial images or point cloud data of a wide area corresponding to the flight range of the target aircraft.

[0057] The sensor (150) may include either a camera and a LiDAR, or a combination thereof. Sensor data collected through the sensor (150) may be an image and / or a point cloud. The sensor (150) continuously collects sensor data at regular intervals, and the processor (110) may store the sensor data in memory (130) or a storage device (140). Since the sensor (150) is mounted on a target aircraft, the image collected by the sensor (150) may be an image (hereinafter referred to as an 'aerial image') taken from a direction looking down at the ground from the target aircraft.

[0058] In addition, the visual navigation method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and may be recorded on a computer-readable medium.

[0059] The above computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the computer-readable medium may be specially designed and configured for embodiments of the present invention, or they may be known and available to a person skilled in the art of computer software. The computer-readable recording medium may include a hardware device configured to store and execute program instructions. For example, the computer-readable recording medium may be magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; ROM; RAM; flash memory, etc. The program instructions may include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer through an interpreter, etc.

[0060] FIG. 2 is a diagram showing a visual navigation device according to an embodiment of the present invention acquiring an aerial image.

[0061] As described above, the visual navigation device (100) is mounted on the target aircraft. While the target aircraft is flying, the visual navigation device (100) can acquire sensor data (Dt), which is an aerial image, through the built-in sensor (150). The visual navigation device (100) can extract reference data (RD), which is an aerial image of the flight area of ​​the target aircraft, from the storage device (140). Typically, the reference data (RD) corresponds to a wider area than the area corresponding to the sensor data (Dt). The visual navigation device (100) can use a visual language model (VLM) to determine which part of the reference data (RD) the sensor data (Dt), which is an aerial image, corresponds to, and based on the determination result, generate navigation information of the target aircraft (e.g., position, attitude, rate of change of position, rate of change of attitude, etc.). Although the example used was that the sensor data (Dt) is an aerial image, the sensor data (Dt) could also be a point cloud collected by LiDAR. In this case, the reference data (RD) becomes a point cloud covering a wider area than the sensor data (Dt).

[0062] FIG. 3 is a conceptual diagram of a visual navigation method according to an embodiment of the present invention.

[0063] A visual navigation method according to one embodiment of the present invention is performed by a visual navigation device (100) or a visual navigation system (10). For convenience, the visual navigation device (100) is described as performing the visual navigation method.

[0064] The visual navigation device (100) is a device that enables a visual language model (VLM) to recognize flight dynamics models and geospatial information, and enables the visual language model (VLM) to output navigation information of the target aircraft.

[0065] The visual navigation device (100) and visual navigation method according to the present invention are designed for the following purposes.

[0066] First is enhanced versatility.

[0067] To ensure that image-based navigation operates robustly even in various weather conditions such as changes in climate or light intensity, rain, or fog, the device and method according to the present invention are designed to exhibit general-purpose performance in various scenarios by utilizing a Visual Language Model (VLM). Furthermore, various dynamic objects, such as pedestrians or vehicles, are acquired in images obtained from an aircraft. In generating a prompt to be input into the Visual Language Model (VLM), the device and method according to the present invention distinguish between these dynamic objects and static objects, such as roads or buildings, that can be used as landmarks. Additionally, when the Visual Language Model (VLM) searches for the location of the sensor data (ultimately the location of the aircraft) by determining the similarity between reference data and sensor data, the similarity determination is performed based on the static objects that serve as landmarks.

[0068] Second, it is real-time performance enhancement.

[0069] The apparatus and method according to the present invention introduces visual odometry and a delayed Kalman filter to overcome the limitation that a visual navigation device (100) cannot operate in real time from the time of prompt input (or transmission) to the time of response output (or reception) when utilizing a visual language model (VLM). The apparatus and method according to the present invention combines visual odometry and a delayed Kalman filter to complement real-time performance by predicting and compensating for navigation information of a target aircraft using visual odometry during the processing time (from the time of prompt input / transmission to the time of response output / reception) of a prompt (which may include sensor data such as a query, status information, and aerial images) transmitted to the visual language model (VLM).

[0070] Third, the introduction of the 3D reconstruction process.

[0071] Since it is advantageous to perform image navigation by accumulating continuous images acquired according to the position change of a target aircraft rather than performing image navigation with a single aerial image, the apparatus and method according to the present invention improve navigation performance by utilizing techniques that generate sub-keyframes, such as Neural Radiation (NeRF) and Simultaneous Localization and Mapping (SLAM), based on a sequential dataset. Furthermore, the apparatus and method according to the present invention can estimate the relationship between an image and the actual environment, such as the correspondence between the pixel coordinate system of an aerial image and the Earth coordinate system of the actual environment (e.g., latitude, longitude), through a 3D reconstruction process that generates a 3D map using sub-keyframes. For example, the correspondence can have precision in meters.

[0072] Fourth, the utilization of flight dynamics.

[0073] Recently, there have been reports that the mathematical and mechanical knowledge of large language models has improved to the level of a typical university student. Therefore, the device and method according to the present invention do not input only aerial images into the visual language model (VLM), but rather transmit state information of the aircraft received from an inertial sensor or flight control module to the visual language model (VLM) along with a timestamp, thereby inducing the visual language model (VLM) to interpret this state information from the perspective of a person controlling a flight while feeling inertia. In addition, since the inference results may vary depending on the level of the model for the target aircraft, the visual navigation device (100) inputs the names or formulas of flight dynamic models, such as a precision model, a simplified model, a kinematic model, and a dynamic model, into the visual language model (VLM) by including them in the prompt during the initialization phase of the visual language model (VLM). And, when the visual navigation device (100) includes state information of the target aircraft in the prompt subsequently transmitted to the visual language model (VLM), the visual language model (VLM) processes the state information of the target aircraft by reflecting the flight dynamics model information (name or formula) described above to generate navigation information (e.g., position, attitude, rate of change of position, rate of change of attitude) regarding the target aircraft.

[0074] Hereinafter, with reference to FIGS. 3 to 8, a visual navigation device (100) and a visual navigation method according to an embodiment of the present invention will be described.

[0075] FIG. 4 is a flowchart illustrating a visual navigation method according to an embodiment of the present invention.

[0076] Referring to FIG. 4, a visual navigation method according to one embodiment of the present invention comprises steps S210 to S270. The visual navigation method illustrated in FIG. 4 is according to one embodiment, and the steps of the visual navigation method according to the present invention are not limited to the embodiment illustrated in FIG. 4 and may be added, changed, or deleted as necessary.

[0077] Step S210 is the sequential dataset generation step.

[0078] The processor (110) of the visual navigation device (100) continuously collects sensor data (Dt) using a sensor (150) mounted on a target aircraft and accumulates the sensor data (Dt) in chronological order to generate a sequential dataset (SD) containing multiple sensor data (Dt) and a timestamp for each sensor data (Dt).

[0079] FIG. 5 is a diagram regarding the acquisition of an aerial image, and FIG. 6 is a diagram explaining the process of estimating the position of an aircraft by inputting aerial images and reference data into a VLM. When a camera is included in the sensor (150) of the visual navigation device (100), the visual navigation device (100) can acquire sensor data (Dt), which is an aerial image such as FIG. 5 and FIG. 6 (a), using the camera. For reference, FIG. 6 (b) is reference data (RD) regarding a flight area extracted from a storage device (140) by the visual navigation device (100), and FIG. 6 (c) shows the position range (Lt) of the target aircraft estimated by the visual language model (VLM) on the reference data (RD).

[0080] Step S220 is the prompt generation step.

[0081] The processor (110) generates a prompt to request navigation information from the visual language model (VLM).

[0082] The processor (110) may include in the prompt any one or a combination thereof among a query requesting navigation information, state information (VSt) of the target aircraft, the name of a dynamic model (flight dynamics model, 'Vehicle model' in FIG. 3) describing the operation of the target aircraft (e.g., Kinematic Model, Dynamics Model, Simplified Model, etc.), the formula of the said dynamic model, sensor data (Dt), and reference data (RD). In this specification, 'Vehicle' is used as a comprehensive term to refer to all means capable of moving on land, sea, and air.

[0083] State information (VSt) may include inertial data of the target aircraft.

[0084] For example, the processor (110) may include the name or formula of the flight dynamics model in the prompt to be delivered to the visual language model (VLM, T1) during the initialization phase of the visual language model (VLM). And, during the step of generating the first navigation information (ND1) using the visual language model (VLM), the processor (110) may include the state information (VSt) of the target aircraft obtained from an inertial sensor included in the sensor (150) or a flight control module mounted on the target aircraft, along with a query text requesting navigation information of the target aircraft, a sequential dataset (SD) (or sensor data (Dt)), and reference data (RD), in the prompt to be delivered to the visual language model (VLM).

[0085] Step S230 is the first navigation information generation step using VLM.

[0086] The processor (110) of the visual navigation device (100) extracts reference data (RD) from the reference database of the storage device (140) and inputs a query (QRY), a sequential dataset (SD), status information (VSt), and reference data (RD) into a visual language model (VLM, T1) to generate first navigation information (ND1). That is, the processor (110) inputs a prompt containing the query (QRY), status information (VSt), reference data (RD), and sequential dataset (SD) into the visual language model (VLM, T1) to generate first navigation information (ND1).

[0087] The visual language model (VLM, T1) may be stored in the storage device (140) of the visual navigation device (100) and then loaded into memory (130), or it may be stored in the VLM server (50). If the visual language model (VLM, T1) stored in the VLM server (50) is used, the processor (110) transmits a prompt to the VLM server (50) or receives a response from the VLM server (50) through the communication device (120).

[0088] In step S230, the processor (110) may perform initialization of the visual language model (VLM). For example, before requesting the first navigation information (ND1) from the visual language model (VLM), the processor (110) may input reference data (RD) as a prompt into the visual language model (VLM) in advance, along with the name and formula of the flight dynamics model (dynamics model) that the visual language model (VLM) wishes to utilize. The flight dynamics model (dynamics model) may be a dynamics model capable of describing the operation of the target aircraft.

[0089] In this specification, navigation information may include position information, pose information, velocity information, acceleration information of a target aircraft, and rates of change for each. The pose information may include the roll, pitch, and yaw of the target aircraft and rates of change for each.

[0090] FIG. 7 is an example diagram of a query (QRY) input into a visual language model (VLM) and the response (RES) of the visual language model (VLM) to it. In FIG. 7, the prompt input into the visual language model (VLM) includes a query (QRY), a first image, and a second image. Here, the first image corresponds to reference data (RD), which is a high-resolution aerial photograph, and the second image corresponds to sensor data (Dt) or a sequential dataset (SD).

[0091] As illustrated in FIG. 7, the visual navigation device (100) requests the visual language model (VLM, T1) to estimate the coordinates of the second image based on reference data (RD). In this process, the visual navigation device (100) also transmits the first image and the second image to the visual language model (VLM) as visual prompts, along with a text query. As another example, the visual navigation device (100) may transmit the reference data (RD) to the visual language model (VLM) during the initialization phase of the visual language model (VLM).

[0092] And, as illustrated in FIG. 7, the visual language model (VLM) transmits the location estimation result of the second image to the visual navigation device (100) through a response (RES) to a query (QRY).

[0093] FIG. 8 is a diagram illustrating reference data and aerial images input into the VLM and the output values ​​of the VLM for the same. In FIG. 8, the area occupied by sensor data (Dt), which is an image captured during flight from the target aircraft, in the reference data (RD) is indicated. Also, as a result of inference by the visual language model (VLM), the estimated position range (Lt) of the target aircraft is indicated in the reference data (RD).

[0094] Step S240 is the second navigation information generation step using visual odometry.

[0095] The processor (110) generates second navigation information (ND2) using visual odometry (T2) based on a sequential dataset (SD).

[0096] Step S240 can be executed in conjunction with Step S230. That is, Step S240 can be performed until the processor (110) inputs a prompt into the visual language model (VLM) and obtains the first navigation information (ND1) from the visual language model (VLM).

[0097] Step S250 is the 3D map generation step.

[0098] The processor (110) generates a three-dimensional map (MP) using a sub-keyframe (SKF) generation technique based on a sequential dataset (SD) (T3). The sub-keyframe generation technique may be Neural Radiation (NeRF) or SLAM.

[0099] Step S260 is the third navigation information generation step using a delayed Kalman filter.

[0100] The processor (110) inputs the first navigation information (ND1) and the second navigation information (ND2) into the delayed Kalman filter (T4) to generate the third navigation information (ND3), which is the navigation information of the target aircraft at the current time. That is, the processor (110) causes the delayed Kalman filter (T4) to estimate the third navigation information (ND3), which is the navigation information of the target aircraft at the current time, based on the first navigation information (ND1) and the second navigation information (ND2), which have different judgment reference times.

[0101] In the process of generating third navigation information (ND3), the processor (110) collects inertial data of the target aircraft from the sensor (150) and inputs the inertial data into the delayed Kalman filter (T4), thereby enabling the delayed Kalman filter (T4) to estimate the third navigation information (ND3) more accurately.

[0102] Additionally, the processor (110) can perform a safety evaluation for each navigation information based on the distribution of the first navigation information (ND1), the second navigation information (ND2), and the third navigation information (ND3) using a delayed Kalman filter (T4). The result of the safety evaluation (SE) for each navigation information can be used as the reliability of each navigation information. That is, the processor (110) can exclude navigation information (ND1 or ND2) with a reliability lower than a predetermined threshold and allow the target aircraft to operate.

[0103] Step S270 is the step of fusing third navigation information with a three-dimensional map.

[0104] The processor (110) can fuse the third navigation information (ND3) with the three-dimensional map (MP) generated in step S250. In this process, the third navigation information (ND3) can be converted to an Earth coordinate system, and the scale of the coordinate system of the target aircraft can be adjusted in comparison with the three-dimensional map. The processor (110) can correlate the third navigation information (ND3) with the three-dimensional map to estimate the relationship between sensor data (Dt) represented as a pixel image and the actual environment.

[0105] The aforementioned visual navigation method has been described with reference to the flowchart presented in the drawings. For simplicity of explanation, the method has been illustrated and described in a series of blocks; however, the present invention is not limited to the order of said blocks, and some blocks may occur in a different order or simultaneously with other blocks as illustrated and described herein, and various other branches, flow paths, and sequences of blocks that achieve the same or similar results may be implemented. Furthermore, not all illustrated blocks may be required for the implementation of the method described herein.

[0106] Meanwhile, in the description with reference to FIG. 4, each step may be further divided into additional steps or combined into fewer steps, depending on an embodiment of the present invention. Additionally, some steps may be omitted as necessary, and the order between steps may be changed. Furthermore, even if other omitted details are omitted, other drawings of this specification and the descriptions made with reference thereto may apply to the content of FIG. 4. Also, the content of FIG. 4 may apply to other parts of this specification.

[0107] Although the present invention has been described above with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as described in the following claims. Explanation of the symbols

[0108] 10: Visual Navigation System 50: VLM Server 100: Visual navigation device 110: Processor 120: Communication device 130: Memory 140: Storage device 150: Sensor 160: Bus

Claims

Claim 1 A visual navigation method comprising: a step in which a visual navigation device continuously collects sensor data using a sensor mounted on a target aircraft and generates a sequential dataset based on said sensor data; a step in which the visual navigation device generates a prompt for requesting navigation information from a visual language model; and a step in which the visual navigation device extracts reference data from a reference database and inputs said prompt, said sequential dataset, and said reference data into said visual language model to generate first navigation information. Claim 2 A visual navigation method according to claim 1, further comprising: a step in which the visual navigation device generates second navigation information using visual odometry based on the sequential dataset; and a step in which the first navigation information and the second navigation information are input into a delayed Kalman filter to generate third navigation information, which is navigation information of a target aircraft at a current time. Claim 3 A visual navigation method according to claim 1, wherein the visual navigation device further comprises the step of generating a three-dimensional map using a sub-keyframe generation technique based on the sequential dataset. Claim 4 A visual navigation method according to paragraph 2, wherein the step of generating the third navigation information further comprises the visual navigation device collecting inertial data of a target aircraft using the sensor and inputting the inertial data into the delayed Kalman filter. Claim 5 A visual navigation method according to claim 1, wherein the step of generating the prompt comprises the visual navigation device including at least one of the state of a target aircraft, the name of a dynamic model describing the operation of the target aircraft, and the formula of the dynamic model in the prompt. Claim 6 A visual navigation device comprising: a memory for storing computer-readable commands; and at least one processor implemented to execute said commands, wherein the at least one processor, by executing said commands, continuously collects sensor data using a sensor mounted on a target aircraft, generates a sequential dataset based on said sensor data, extracts reference data from a reference database, generates a prompt for requesting navigation information from a visual language model including said reference data and said sequential dataset, and inputs said prompt into the visual language model to generate first navigation information. Claim 7 A visual navigation device according to claim 6, wherein at least one processor is configured to generate second navigation information using visual odometry based on the sequential dataset, and to generate third navigation information, which is navigation information of a target aircraft at the current time, by inputting the first navigation information and the second navigation information into a delayed Kalman filter. Claim 8 A visual navigation device according to claim 6, wherein at least one processor is configured to generate a three-dimensional map using a sub-keyframe generation technique based on the sequential dataset. Claim 9 A visual navigation device according to claim 7, wherein at least one processor is configured to collect inertial data of a target aircraft using the sensor and input the inertial data into the delayed Kalman filter during the process of generating the third navigation information. Claim 10 A visual navigation device according to claim 6, wherein the at least one processor is configured to include in the prompt at least one of the state of the target aircraft, the name of a dynamic model describing the operation of the target aircraft, and the formula of the dynamic model in the prompt during the process of generating the prompt. Claim 11 A visual navigation device according to claim 7, wherein at least one processor is configured to generate the second navigation information using visual odometry based on the sequential dataset from the point in time when the prompt, the sequential dataset, and the reference data are input into the visual language model, until the point in time when an output is obtained from the visual language model. Claim 12 A visual navigation system comprising: a first memory, a first processor, and a storage device, a VLM server that runs a visual language model (VLM) in the storage device; and a visual navigation device, wherein the visual navigation device comprises: a second memory that stores computer-readable commands; and a second processor implemented to execute the commands, wherein the second processor is configured to continuously collect sensor data using a sensor mounted on a target aircraft by executing the commands, generate a sequential dataset based on the sensor data, extract reference data from a reference database, transmit a prompt including the reference data and the sequential dataset to the VLM server via wireless communication, receive a response to the prompt from the VLM server, and generate first navigation information including position information and attitude information of the target aircraft based on the response.

Images