Autonomous flight method using visual navigation in event of GNSS anomaly

Abstract

The present invention relates to a method that switches a drone to flight using visual navigation in a situation in which it is impossible to use a GNSS signal, wherein a camera is zoom-controlled according to the altitude, and a correction coefficient dependent on the zoom magnification can be applied to optical flow calculation for visual navigation to achieve autonomous flight through accurate direction identification and velocity calculation.

Description

An automated flight method using visual navigation in GNSS anomaly situations.

[0001] The present invention relates to a method for enabling a drone to fly automatically in a situation where a GNSS signal is unavailable, and more specifically, to a method for enabling a drone to fly using visual navigation in a situation where a GNSS signal is unavailable, controlling a camera zoom according to altitude, and applying a correction factor according to the zoom ratio to optical flow calculation for visual navigation, thereby enabling automatic flight through accurate direction confirmation and speed calculation.

[0002]

[0003] Typically, most drones are equipped with GNSS modules, such as GPS receivers, which allow them to fly precisely and minimize positional errors by using real-time location information that can be verified.

[0004] Therefore, if there is a problem with the GNSS module or an external attempt is made to jam the satellite signal, the location information becomes unavailable, making it very difficult for the drone to fly accurately.

[0005] For this reason, it may be said that a means is needed to protect drones used for specific missions such as reconnaissance or photography from abnormal situations such as radio interference, and according to that need, inventions such as “Hacking Response System for Unmanned Aerial Vehicles” in Korean Patent Publication No. 10-2020-0016432 and “Automatic Flight Device for Responding to Situations Where GPS Signal Reception Is Unable” in Korean Patent Publication No. 10-2022-0014179 have been proposed and published.

[0006] That is, the “Hacking Response System for Unmanned Aerial Vehicle” of the above-mentioned Republic of Korea Patent Publication No. 10-2020-0016432 proposes an invention regarding a hacking response system that analyzes GPS signals received in real time to detect and remove fake signals, thereby responding to spoofing hacking attempts, and also responding to jamming hacking attempts by flying along preset flight path information when reception of all signals is blocked.

[0007] In addition, in the above-mentioned Korean Patent Publication No. 10-2022-0014179 entitled “Automatic flight device for responding to a situation in which GPS signal reception is impossible,” an invention is proposed regarding an automatic flight device that enables an unmanned aerial vehicle to safely autonomously fly and land toward a safe area by calculating using the last collected location information, coordinate information for each of a plurality of preset safe areas, and the heading value of the unmanned aerial vehicle in the event of a failure in GPS signal reception.

[0008] By utilizing the above-mentioned prior inventions, it is expected that it will be possible to respond to abnormal situations such as radio interference that may occur during drone flight, to the extent that it can prevent the drone from crashing or being captured.

[0009] However, the above-mentioned prior inventions only present comprehensive concepts for responding to abnormal situations such as radio interference, and have structural shortcomings in that they do not actually suggest specific methods for precisely flying a drone to its original destination or a safe location without satellite signals.

[0010] In other words, it can be said that it is difficult to fully implement the technology to prevent drones from crashing or being captured in abnormal situations using only the above-mentioned prior art inventions, and for this reason, it can be said that there is a need for a new invention that suggests a method to precisely fly a drone to its original destination or a safe location without a satellite signal.

[0011]

[0012] The present invention has a structural defect in that the above-mentioned inventions only present a comprehensive concept for responding to abnormal situations such as radio interference that may occur during the flight of a drone, and do not actually propose a specific method for precisely flying a drone to its original destination or a safe location without a satellite signal. Therefore, the purpose of the present invention is to present a solution to this problem.

[0013] To elaborate, in the case of flying only with traditional navigation, inertial navigation, in abnormal situations such as radio interference that may occur during drone flight, there is a limitation that cannot be resolved by continuously accumulating errors.

[0014] In addition, the image matching technique utilizing satellite maps is not only unable to respond to changes in the ground condition that occur over time based on the time of shooting the satellite map, but also has limitations in that it cannot be used in wide areas such as deserts without specific patterns.

[0015] Accordingly, since there is no suitable countermeasure for abnormal situations such as radio interference using conventional methods, including the above-described inventions, the present invention aims to provide a solution thereto.

[0016]

[0017] The present invention aims to achieve the above-mentioned purpose,

[0018] The present invention proposes an automatic flight method using visual navigation in a GNSS abnormality situation, comprising: an abnormality detection step for detecting an abnormality in a GNSS mounted on a drone; a shooting step for adjusting the shooting direction of a camera mounted on the drone to a vertical downward direction when an abnormality in the GNSS is detected and shooting a real-time image; an altitude confirmation step for confirming the current flight altitude using a sensor mounted on the drone when an abnormality in the GNSS is detected; a zoom control step for controlling the zoom of the camera according to the confirmed flight altitude; a calculation step for tracking the positional changes of pixels in an image using an optical flow method while the zoom of the camera is controlled to confirm the direction in which the drone is flying and at the same time calculating the speed at which the drone is flying by reflecting the current flight altitude and zoom magnification; and a flight control step for controlling the flight of the drone using an inertial navigation method using the flight direction and speed of the drone confirmed and calculated in the calculation step.

[0019] At this time, the present invention is characterized in that it calculates the sunrise and sunset times on the day of the flight as soon as an abnormality in GNSS is detected, divides the current time into daytime or nighttime based on the calculated sunrise and sunset times, and independently adjusts the shooting direction of the thermal imaging camera mounted on the drone to vertically downward at night, thereby shooting real-time thermal imaging images used in the calculation stage.

[0020] In addition, the present invention is characterized in that it is configured to eliminate or minimize the occurrence of a calculation value error according to the flight altitude of the drone by applying a correction coefficient according to the flight altitude of the drone and the current zoom ratio to the optical flow calculation using a preset correction formula.

[0021]

[0022] The automatic flight method using visual navigation in a GNSS anomaly situation according to the present invention is as follows:

[0023] In a situation where GNSS signals are unavailable, the drone switches to flying using visual navigation, but the camera zoom is controlled according to altitude, and a correction factor according to the zoom ratio is applied to the optical flow calculation for visual navigation, which results in the effect of realizing automatic flight based on accurate direction confirmation and speed calculation.

[0024] In addition, the present invention is configured to distinguish between daytime and nighttime when GNSS signals become unavailable, and to use a digital camera during the daytime and a thermal imaging camera at night, so that automatic flight by visual navigation can be realized without problems even at night.

[0025]

[0026] Figure 1 is a basic flowchart of an automatic flight method using visual navigation in a GNSS anomaly situation according to the present invention.

[0027] Figure 2 is a flowchart showing a current point-of-view distinction process for selecting a shooting camera in the shooting stage included in the present invention.

[0028] Figure 3 is a flowchart showing the zoom control signal generation process in the altitude confirmation step included in the present invention.

[0029] Figure 4 is a flowchart showing the progress of the calculation step and flight control step included in the present invention.

[0030]

[0031] The automatic flight method using visual navigation in a GNSS anomaly situation according to the present invention is as follows:

[0032] The present invention is characterized by comprising: an abnormality detection step for detecting an abnormality in a GNSS mounted on a drone; a shooting step for adjusting the shooting direction of a camera mounted on the drone to vertically downward when an abnormality in the GNSS is detected and shooting a real-time image; an altitude confirmation step for confirming a current flight altitude using a sensor mounted on the drone when an abnormality in the GNSS is detected; a zoom control step for controlling the zoom of the camera according to the confirmed flight altitude; a calculation step for confirming the direction in which the drone is flying by tracking the change in the position of pixels for the image using an optical flow method while the zoom of the camera is controlled, and calculating the speed at which the drone is flying by reflecting the current flight altitude and zoom magnification; and a flight control step for controlling the flight of the drone using an inertial navigation method using the flight direction and speed of the drone confirmed and calculated in the calculation step.

[0033]

[0034] The present invention relates to a method for enabling a drone to fly automatically in a situation where GNSS signals are unavailable.

[0035] It is characterized in that it comprises an abnormality detection step (S100) for detecting an abnormality in GNSS mounted on a drone; a shooting step (S110) for adjusting the shooting direction of a camera mounted on the drone to vertically downward when an abnormality in GNSS is detected and shooting a real-time image; an altitude confirmation step (S120) for checking the current flight altitude using a sensor mounted on the drone when an abnormality in GNSS is detected; a zoom control step (S130) for controlling the zoom of the camera according to the confirmed flight altitude; a calculation step (S140) for tracking the positional changes of pixels for the image using an optical flow method while the zoom of the camera is controlled to check the direction in which the drone is flying and at the same time calculating the speed at which the drone is flying by reflecting the current flight altitude and zoom magnification; and a flight control step (S150) for controlling the flight of the drone using an inertial navigation method using the flight direction and speed of the drone checked and calculated in the calculation step (S140).

[0036]

[0037] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0038] First, as illustrated in Fig. 1, the above-described abnormality detection step (S100) is a step that marks the beginning of automatic flight control by image navigation proposed in the present invention, and is based on the point in time at which an abnormality in a GNSS mounted on a drone having multiple rotary wings is detected.

[0039] That is, a drone that can be used for the purpose of performing missions such as filming is equipped with a GNSS (Global Navigation Satellite System) module such as a GPS receiver for the purpose of receiving satellite signals. In the above-mentioned abnormality detection step (S100), an abnormality in the GNSS module itself or an abnormality in the satellite signal received by the GNSS module is detected so that a response procedure can be initiated.

[0040] Therefore, an analysis device (100) or analysis system for detecting anomalies in the GNSS module itself or in satellite signals received by the GNSS module must be provided in the main body of the drone or in a ground control device, and it is desirable that it be always kept activated during the flight of the drone.

[0041] In addition, it is preferable that the above analysis device (100) or analysis system be configured to detect not only attempts at spoofing but also attempts at jamming of a drone. Among these, detection methods for spoofing may include a method for monitoring the sensitivity of a satellite signal, a method for analyzing the quality of a satellite signal, a method for detecting frequency interference, etc., but other methods may be used if they are effective in detection.

[0042] And the above analysis device (100) or analysis system must generate an emergency control signal for response immediately upon detecting an attempt at spoofing or jamming of the drone, and transmit the generated control signal to a control device (110) provided in the main body of the drone or a control device (110) provided in a ground control device.

[0043] At this time, the control device (110) may be a device that controls the camera alone or a multi-functional device that can control the flight of the drone and simultaneously control the camera mounted on the drone, and as soon as an emergency control signal is received, the device must control the camera alone or control the drone and the camera simultaneously, giving priority to other signals.

[0044]

[0045] In addition, as illustrated in FIG. 1, the above-described shooting step (S110) is a step performed immediately after an abnormality in GNSS is detected, and adjusts the shooting direction of the camera mounted on the drone to vertically downward, thereby allowing real-time video to be captured, thereby ensuring that real-time video information, which serves as the basis for video navigation, can be secured.

[0046] That is, when an abnormality in GNSS is detected in a flight state without video recording of the drone, the control device (110) immediately switches the camera to the shooting mode upon receiving an emergency control signal and maintains or adjusts the shooting direction to a vertical downward direction so that real-time video can be captured.

[0047] In addition, when an abnormality in GNSS is detected during a flight state involving video recording of the drone, the control device (110) immediately maintains or adjusts the shooting direction of the camera to a vertical downward direction upon receiving an emergency control signal so that real-time video can be recorded.

[0048] At this time, the emergency control signal received by the control device (110) must have priority over other control signals, so image capture by the previous control signal must be stopped immediately.

[0049] Accordingly, in the above shooting step (S110), an image of the vertical downward direction of the drone in flight can be captured in real time, and image information corresponding thereto is generated and transmitted to a designated device immediately after generation, so that it can be used to confirm the direction in which the drone is flying and to calculate the speed at which it is flying in a subsequent step.

[0050] However, the above configuration is only meaningful during the day when it is possible to capture images clear enough to clearly identify terrain features below the drone using a general digital camera. The image information generated at night when it is difficult or impossible to capture clear images is difficult or impossible to use in the process of confirming the direction in which the drone is flying and calculating the speed at which it is flying.

[0051] As a solution to the above problem, the present invention is characterized in that, when an abnormality in GNSS is detected, the current time can be distinguished as daytime or nighttime based on the sunrise and sunset times on the day the drone is flying, and when the current time when the distinction is completed is nighttime, a thermal imaging camera is used to capture an image.

[0052] To this end, the drone must be equipped with a digital camera capable of taking general images along with a gimbal, as well as a thermal imaging camera capable of taking thermal images along with a gimbal, and the two types of cameras mounted can be controlled simultaneously or selectively by the control device (110).

[0053] More specifically, as illustrated in FIG. 2, the analysis device (100) or the analysis system can calculate the sunrise and sunset times of the day based on the date and time at which it detects an abnormality in the GNSS mounted on the drone or detects an attempt to jam the drone, and can distinguish the current time as daytime or nighttime based on the calculated sunrise and sunset times.

[0054] At this time, in order to calculate the sunrise and sunset times, it is necessary to know the current date and time as well as the current latitude and longitude based on the drone, which can be confirmed through the analysis device (100) or analysis system.

[0055] Here, a method for confirming the current latitude and longitude may be a method using latitude and longitude information confirmed through the GNSS module immediately before an abnormality is detected in the above-mentioned abnormality detection step (S100), but there is no limitation to this method.

[0056] And, using the following code inputted into the above analysis device (100) or analysis system, the sunrise and sunset times on the day the drone is flying can be calculated. (The codes listed in Table 1 below are examples of codes that can calculate sunrise and sunset times and can be used as is, but some of them can be changed or other commands can be added and used as needed.)

[0057] int M = Month;int D = Day;double latitude = Lat;double longitude = Lon;double timeSpan = (M - 3) * 30 + (D - 21);if (timeSpan < 0) timeSpan = 365;double axialTilt = 23.5 * Math.sin(D2R * ((360.0 * timeSpan) / 365.0));double theta3Rad = Math.acos(Math.tan(D3R * axialTilt) * Math.tan(D2R * latitude));double ratio = 2 * R2D * theta3Rad / 360 - 0200625;double sunrise = 60 * 60 * 24 * ratio / 2 + 60 * (135 - longitude) * 4;double sunset = 60 * 60 * 12 + 60 * 60 * 24 * (1 - ratio) / 2 + 60 * (135 - longitude) * 4;if((sunrise <S)&&(S<sunset)) return true;else return false;

[0058] Thereafter, when the current time confirmed as a result of the calculation is daytime, a control signal targeting the digital camera mounted on the drone is generated and transmitted to the control device (110), thereby enabling simultaneous vertical downward adjustment of the camera and shooting of an image downward.

[0059] However, when the current time confirmed as a result of the calculation is night, a control signal targeting the thermal imaging camera mounted on the drone is generated and transmitted to the control device (110), thereby enabling simultaneous vertical downward adjustment of the thermal imaging camera and shooting of a thermal image downward.

[0060] In this process, since there is no need to stop video shooting using a digital camera, a situation may arise where two types of video information are generated simultaneously. However, since the video information generated by the digital camera is not generated for the purpose of calculating the drone's flight speed, only real-time thermal images can be used in the subsequent calculation step (S140).

[0061] As a result, when the above shooting step (S110) is performed at night, a thermal image of the vertical downward direction of the drone in flight can be captured in real time, and the image information is generated and transmitted to a designated device, so that it can be used to confirm the direction in which the drone is flying and to calculate the speed at which it is flying in a subsequent step.

[0062]

[0063] In addition, as illustrated in FIG. 1, the altitude confirmation step (S120) is also a step performed immediately after an abnormality in GNSS is detected, and uses DTED (Digital Terrain Elevation Data) or LiDar, etc. to enable the altitude of the actual flight location based on the ground, rather than the AGL (Above Ground Level) based on the takeoff point, to be confirmed.

[0064] Accordingly, the radio altimeter, barometric pressure sensor or lidar equipped on the drone can generate measurement information at all times or from the time an abnormality in GNSS is detected or a jamming attempt is detected, and the generated measurement information can be transmitted to an analysis device (100) equipped on the drone or to an analysis device (100) equipped on a ground control device and used in the process of generating a control signal for controlling the zoom of the camera.

[0065] That is, as illustrated in FIG. 3, the analysis device (100) or the analysis system can apply distance information or atmospheric pressure numerical information included in the measurement information to the input DTED or calculate the current altitude value using LiDar or the like, and can generate a zoom control signal for zoom control of a digital camera or thermal imaging camera based on the altitude value.

[0066] In addition, the zoom control step (S130) illustrated in FIG. 1 is a step that is performed immediately after a zoom control signal for controlling the zoom of the camera is generated or includes a process for generating the zoom control signal, and enables more accurate image analysis for image navigation to be realized by controlling the zoom of the camera according to the confirmed flight altitude.

[0067] That is, in order to realize video navigation, it must be accompanied by image analysis using the optical flow method that tracks the change in the position of pixels in the image captured by the camera. However, this optical flow method of image analysis has a problem in that the resolution and precision of the image decrease as the altitude of the drone increases, so the accuracy of the analysis decreases or the problem occurs in that analysis data that can be used for video navigation cannot be generated.

[0068] Accordingly, the present invention allows the control device (110) to perform zoom control of the camera according to the flight altitude of the drone based on the zoom control signal generated by the analysis device (100) or the analysis system, thereby producing the effect of enabling accurate realization of image analysis using the optical flow method even when the drone is flying at a high altitude of 100 m or higher.

[0069] As one embodiment of the present invention regarding the above zoom control step (S130), the analysis device (100) or the analysis system can generate a zoom control signal to cause the camera to zoom out to the maximum at an altitude of 15 m or less based on a preset maximum altitude of 150 m, can generate a zoom control signal to cause the camera to zoom in by 10% based on the maximum value at an altitude exceeding 15 m but less than or equal to 30 m, and can generate a zoom control signal to cause the camera to zoom in to the maximum at an altitude exceeding 135 m but less than or equal to 150 m by maintaining the same method as above.

[0070] At this time, the maximum altitude in the above-described embodiment is not a fixed value because it depends on the performance of the camera mounted on the drone, and a different maximum altitude may be applied depending on the product.

[0071]

[0072] In addition, as illustrated in FIG. 1, the calculation step (S140) is a step performed immediately after the zoom control of the camera by the control device (110) is performed, and enables more accurate image analysis for image navigation of the drone to be realized using the optical flow method described above.

[0073] That is, in the calculation step (S140), the analysis device (100) or the analysis system tracks the change in the position of pixels in the image using an optical flow method while the zoom of the camera is controlled to determine the direction in which the drone is flying, and at the same time, calculates the speed at which the drone is flying by reflecting the current flight altitude and zoom magnification.

[0074] However, in the above zoom control step (S130), a zoom control signal is generated based only on the altitude of the drone, so there is a problem that the speed value confirmed by the optical flow method and the actual speed of the drone are significantly inconsistent.

[0075] This is because the FOV (Field of View) change value is formed nonlinearly during the process of the camera zooming in or out in the above zoom control step.

[0076] That is, when checking the change in FOV for one product among digital cameras, the FOV value at the maximum zoom out state was 58.92695°, the FOV value at the 10% zoom in state based on the maximum value was 51.74155°, the FOV value at the 20% zoom in state was 44.09145°, the FOV value at the 30% zoom in state was 38.06503°, the FOV value at the 40% zoom in state was 32.60511°, the FOV value at the 50% zoom in state was 26.98874°, the FOV value at the 60% zoom in state was 21.23717°, the FOV value at the 70% zoom in state was 16.49911°, the FOV value at the 80% zoom in state was 13.4017°, and the 90% The FOV value in the zoomed-in state was confirmed to be 10.28449°, and the FOV value in the maximum zoomed-in state was confirmed to be 7.437238°.

[0077] In this way, since the FOV (Field of View) change value of the camera mounted on the drone is formed nonlinearly, image analysis using the optical flow method that simply reflects the current flight altitude and zoom ratio is bound to be inaccurate.

[0078] Therefore, it is desirable for the above analysis device (100) or analysis system to reflect a correction coefficient in the input correction formula so that an accurate calculation value can be derived through image analysis, and when a zoom control signal to which the above correction formula is applied is generated, the speed value confirmed by the optical flow method and the actual speed of the drone can be almost identical or the same.

[0079] That is, as illustrated in FIG. 4, the calculation step (S140) is characterized in that it is configured to eliminate or minimize the occurrence of calculation value errors according to the drone's flight altitude by applying a correction coefficient according to the drone's current flight altitude and zoom ratio to the optical flow calculation using a pre-input correction formula.

[0080] Meanwhile, in a situation where the drone is flying at an altitude higher than the maximum altitude, only the altitude changes while the camera mounted on the drone is already zoomed in to the maximum, and since this is a linear variable, the optical flow calculation can be performed by dynamically reflecting only the current altitude to a fixed zoom ratio value.

[0081] In addition, as illustrated in FIG. 1, the flight control step (S150) is a step performed immediately after image analysis using the optical flow method is performed, and enables inertial navigation to be realized based on the flight direction and speed of the drone confirmed and calculated through image analysis.

[0082] That is, in the above flight control step (S150), the control device (110) controls the flight of the drone using an inertial navigation method that uses direction data and speed data, which are analysis data generated by the analysis device (100) or the analysis system.

[0083] Accordingly, the drone can continue to fly along the original flight path or continue to fly along a new flight path even in a situation where an abnormality is detected in the GNSS module, a spoofing attempt occurs, an abnormality is detected in the satellite signal received by the GNSS module, or a jamming attempt is detected.

[0084] Afterwards, except in the case where a problem occurs in the GNSS module, when the drone leaves the range where spoofing or jamming is attempted, the GNSS signal can be received normally again, and in this case, the flight control method for the drone can be switched to the original method using the GNSS signal.

[0085]

[0086] The embodiments introduced above are provided as examples so that the technical idea of ​​the present invention can be sufficiently conveyed to a person having ordinary skill in the art to which the present invention pertains, and the present invention is not limited to the embodiments described above and may be embodied in other forms.

[0087] Additionally, identical reference numbers throughout the specification represent identical components.

[0088]

[0089] The automatic flight method using visual navigation in a GNSS anomaly situation according to the present invention is as follows:

[0090] It has sufficient potential for industrial use, as it enables automatic flight by accurately confirming direction and calculating speed in situations where drones cannot use GNSS signals, and it also enables automatic flight by visual navigation without problems even at night.

Claims

1. Anomaly detection step (S100) for detecting anomalies in the GNSS mounted on the drone; When an abnormality in GNSS is detected, the shooting direction of the camera mounted on the drone is adjusted vertically downward, and a shooting step (S110) is performed to shoot real-time images; When an abnormality in GNSS is detected, an altitude confirmation step (S120) is performed to confirm the current flight altitude using a sensor mounted on the drone; A zoom control step (S130) for controlling the zoom of the camera according to the confirmed flight altitude; A calculation step (S140) for tracking the positional changes of pixels in an image using an optical flow method while the zoom of the camera is controlled to determine the direction in which the drone is flying, and at the same time calculating the speed at which the drone is flying by reflecting the current flight altitude and zoom ratio; and An automatic flight method using visual navigation in a GNSS anomaly situation, characterized in that it comprises a flight control step (S150) for controlling the flight of a drone using an inertial navigation method by using the flight direction and speed of the drone confirmed and calculated in the above calculation step (S140); 2. In paragraph 1, When an anomaly in GNSS is detected, Calculate the sunrise and sunset times on the day of your flight, Based on the calculated sunrise and sunset times, the current time is divided into daytime and nighttime. An automatic flight method using image navigation in a GNSS anomaly situation, characterized in that the method is configured to capture real-time thermal images used in the calculation step (S140) by independently adjusting the shooting direction of the thermal imaging camera mounted on the drone to vertically downward in the shooting step (S110) at night.

3. In paragraph 1, In the above calculation step (S140), An automatic flight method using visual navigation in a GNSS anomaly situation, characterized in that the occurrence of a calculation error according to the drone's flight altitude is eliminated or minimized by applying a correction coefficient according to the drone's flight altitude and current zoom ratio to an optical flow calculation using a preset correction formula.

Images