Aerial vehicle for tracking and shooting down illegal aerial vehicle and operation method therefor

Abstract

This operation method of an aerial vehicle, which is performed by at least one aerial vehicle, may comprise the steps of: detecting a target aerial vehicle on the basis of an image captured via an image capturing device; on the basis of an area of the target aerial vehicle in the captured image and a center point within the captured image, controlling the image capturing device to track the target aerial vehicle, and acquiring tracking information of the image capturing device; acquiring pose information of the target aerial vehicle on the basis of the captured image; and on the basis of the tracking information of the image capturing device and the pose information of the target aerial vehicle, generating a flight guidance command for tracking and shooting down the target aerial vehicle.

Description

Aircraft and its operation method for tracking and shooting down illegal aircraft

[0001] It relates to aircraft and how they operate.

[0002] The present invention was derived from research conducted as part of the individual basic research of the Ministry of Science and ICT (MSIT) (Project Unique Number: 1711191634, Project Number: 2021R1G1A1095335, Project Management Agency: National Research Foundation of Korea, Research Project Name: SA-VINS: Development of a Smart Autonomous Navigation System for Aerial Robots for Inventory Management in Indoor Warehouses, Project Performing Agency: Kyungpook National University, Research Period: 2023.03.01 ~ 2024.02.29).

[0003] Meanwhile, the Korean government, which provided the task, has no property interest in any aspect of the present invention.

[0004] With the rapid increase in the use of unmanned aerial vehicles (UAVs) in recent years, there has been a growing number of instances of these aircraft being flown illegally or used for terrorist purposes over airports, military bases, and critical infrastructure. These illegal aircraft pose a serious threat to public safety.

[0005] Therefore, the development of a response system capable of quickly and accurately tracking and shooting down illegal aircraft is required. Prior Art Document: (Patent Document 1) Registered Patent Document No. 10-2308700 (September 28, 2021)

[0006] The goal is to automatically detect illegal aircraft and quickly and accurately track and shoot them down based on tracking information from video recording devices and pose information of the illegal aircraft.

[0007] According to one aspect, a method for operating an aircraft performed by at least one aircraft may include: detecting a target aircraft based on an image captured by an image capturing device; controlling the image capturing device to track the target aircraft based on an area of ​​the target aircraft within the captured image and a center point within the captured image, and obtaining tracking information of the image capturing device; obtaining pose information of the target aircraft based on the captured image; and generating a flight guidance command for tracking and shooting down the target aircraft based on the tracking information of the image capturing device and the pose information of the target aircraft.

[0008] According to one embodiment, the step of controlling the image capturing device may include the step of setting a monitoring area including an area of ​​the target aircraft within the captured image; and the step of controlling the image capturing device so that an error between a center point of the monitoring area and a center point within the captured image is minimized.

[0009] According to one embodiment, the tracking information may include relative altitude information for the target aircraft with respect to the at least one aircraft and relative direction information of the target aircraft with respect to the at least one aircraft.

[0010] According to one embodiment, the step of acquiring pose information of the target aircraft may include: a step of acquiring coordinate information for preset positions of the target aircraft based on a monitoring area including an area of ​​the target aircraft within the captured image; and a step of acquiring speed information including moving speed information and moving direction information of the target aircraft based on the coordinate information with respect to the image capturing device.

[0011] According to one embodiment, the step of obtaining the coordinate information may include the step of calculating a first coordinate representing the upper left coordinate that is the farthest from the center coordinate of the monitoring area; and the step of calculating a second coordinate representing the upper right coordinate that is the farthest from the center coordinate of the monitoring area.

[0012] According to one embodiment, the step of obtaining the pose information may include: a step of calculating a relative angle for estimating a movement direction of the target aircraft based on the first coordinate and the second coordinate; and a step of estimating relative pose information and movement direction information of the target aircraft based on the sign of the relative angle with respect to the image capturing device.

[0013] According to one embodiment, the step of generating the flight guidance command may include the step of determining a predicted movement path of the target aircraft and a point of impact of the target aircraft based on tracking information of the video capturing device and pose information of the target aircraft; and the step of generating velocity vector information for tracking and shooting down the target aircraft based on the predicted movement path of the target aircraft and the point of impact.

[0014] According to one embodiment, the step of generating the flight guidance command may include the step of generating the flight guidance command based on a mode corresponding to the range of the relative angle, when a preset number or more of frames having the same sign of the relative angle are continuously detected from the captured image.

[0015] According to one embodiment, the step of generating the flight guidance command may include: determining the target aircraft as a first mode in which there is no left-right movement when the absolute value of the relative angle satisfies a threshold value; and generating first velocity vector information for tracking and shooting down the target aircraft based on tracking information of the image capturing device according to the first mode.

[0016] According to one embodiment, the step of generating the flight guidance command may include the step of determining the target aircraft as a second mode in which the target aircraft moves left or right when the absolute value of the relative angle exceeds a threshold value; and the step of generating second velocity vector information for tracking and shooting down the target aircraft based on tracking information of the image capturing device and pose information of the target aircraft according to the second mode.

[0017] According to another aspect, an aircraft is provided, comprising: a communication device; an image capturing device; a processor; and a memory storing instructions executable by the processor, wherein the processor, by executing the instructions, detects a target aircraft based on an image captured by the image capturing device, controls the image capturing device to track the target aircraft based on an area of ​​the target aircraft within the captured image and a center point within the captured image, obtains tracking information of the image capturing device, obtains pose information of the target aircraft based on the captured image, and generates a flight guidance command for tracking and shooting down the target aircraft based on the tracking information of the image capturing device and the pose information of the target aircraft.

[0018] According to one embodiment, the processor can set a monitoring area including an area of ​​the target aircraft within the captured image by executing the commands, and control the image capturing device so that an error between a center point of the monitoring area and a center point within the captured image is minimized.

[0019] According to one embodiment, the processor, by executing the commands, obtains coordinate information for preset locations of the target aircraft based on a monitoring area including an area of ​​the target aircraft within the captured image, and obtains speed information including moving speed information and moving direction information of the target aircraft based on the coordinate information with respect to the image capturing device.

[0020] According to one embodiment, the processor can, by executing the instructions, calculate a first coordinate indicating an upper left coordinate that is the farthest from the center coordinate of the monitoring area, and calculate a second coordinate indicating an upper right coordinate that is the farthest from the center coordinate of the monitoring area.

[0021] According to one embodiment, the processor, by executing the instructions, calculates a relative angle for estimating a movement direction of the target aircraft based on the first coordinate and the second coordinate, and estimates relative attitude information and movement direction information of the target aircraft based on the sign of the relative angle with respect to the image capturing device.

[0022] According to one embodiment, the processor can generate the flight guidance command based on a mode corresponding to the range of the relative angle when, by executing the instructions, a preset number or more of frames having the same sign of the relative angle are continuously detected from the captured image.

[0023] According to one embodiment, the processor, by executing the instructions, determines the target aircraft to be in a first mode, which is a first state in which there is no left-right movement, if the absolute value of the relative angle satisfies a threshold value, and according to the first mode, based on the tracking information of the video recording device, generates first velocity vector information for tracking and shooting down the target aircraft.

[0024] According to one embodiment, the processor, by executing the instructions, determines the target aircraft to be in a second mode in which the target aircraft moves left or right when the absolute value of the relative angle exceeds a threshold value, and according to the second mode, generates second velocity vector information for tracking and shooting down the target aircraft based on tracking information of the video capturing device and pose information of the target aircraft.

[0025] According to another aspect, a computer program stored in a computer-readable storage medium for executing a method for operating an aircraft, the method for operating an aircraft, comprising: detecting a target aircraft based on an image captured by an image capturing device; controlling the image capturing device to track the target aircraft based on an area of ​​the target aircraft within the captured image and a center point within the captured image, and obtaining tracking information of the image capturing device; obtaining pose information of the target aircraft based on the captured image; and generating a flight guidance command for tracking and shooting down the target aircraft based on the tracking information of the image capturing device and the pose information of the target aircraft.

[0026] It can automatically detect illegal aircraft and quickly and accurately track and shoot down illegal aircraft based on tracking information from the video recording device and pose information of the illegal aircraft.

[0027] The present disclosure can be readily understood by the combination of the following detailed description and the accompanying drawings, wherein reference numerals refer to structural elements.

[0028] FIG. 1 is a conceptual diagram illustrating an operation for an aircraft to pursue and shoot down a target aircraft, according to one embodiment.

[0029] Figure 2 is a flowchart illustrating a method of operating an aircraft according to one embodiment.

[0030] FIG. 3 is a drawing for explaining a process of controlling an image capturing device to track a target aircraft according to an embodiment.

[0031] Figure 4 is a drawing for explaining the coordinate system of an aircraft according to one embodiment.

[0032] FIG. 5a is a drawing for explaining a process of obtaining tracking information of a target aircraft for an aircraft according to an embodiment, and FIG. 5b is a drawing for explaining a process of calculating velocity vector information of a target aircraft using tracking information of the target aircraft according to an embodiment.

[0033] FIG. 6 is a drawing for explaining a process in which an aircraft performs a flight according to a flight guidance command and tracks a target aircraft, according to an embodiment.

[0034] FIG. 7 is a drawing for explaining a process of acquiring pose information of a target aircraft according to an embodiment.

[0035] FIG. 8 is a drawing for explaining a process of binarizing an image of a target aircraft according to an embodiment.

[0036] FIG. 9 is a drawing for explaining a process of calculating coordinates for feature points of a target aircraft and calculating relative angles, according to an embodiment.

[0037] FIG. 10 is a drawing for explaining a process of obtaining pose information of a target aircraft based on a sign of a relative angle, according to an embodiment.

[0038] FIG. 11 is a drawing for explaining the results of comparing a case in which velocity vector information of a target aircraft is obtained using tracking information of an image capturing device and a case in which velocity vector information of a target aircraft is obtained using tracking information of an image capturing device and pose information of the target aircraft, according to one embodiment.

[0039] FIG. 12 is a diagram for explaining a process of obtaining velocity vector information based on tracking information of an image capturing device and pose information of a target aircraft, according to an embodiment.

[0040] Figure 13 is a block diagram illustrating the configuration of an aircraft according to one embodiment.

[0041] Below, various embodiments are described in detail with reference to the drawings. The embodiments described below may be implemented in various different forms. To more clearly explain the features of the embodiments, detailed descriptions of matters commonly known to those skilled in the art to which the embodiments pertain will be omitted.

[0042] Meanwhile, when a component is said to be "connected" to another component in this specification, this includes not only cases where it is "directly connected" but also cases where it is "connected with another component in between." Furthermore, when a component is said to "include" another component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

[0043] Additionally, terms including ordinal numbers, such as "first" or "second," used herein may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

[0044] As used herein, "aircraft" may refer to a winged vehicle or machine capable of flying. Furthermore, "aircraft" may refer to a manned aircraft or an unmanned aircraft. Furthermore, "aircraft" may refer to an aircraft that is piloted directly on board and controlled by a pilot, or an aircraft that operates autonomously or semi-autonomously through a program on the ground without a pilot on board.

[0045] Hereinafter, specific embodiments of the present invention described above will be described with reference to FIGS. 1 to 13.

[0046] FIG. 1 is a conceptual diagram illustrating an operation of an aircraft (10) to pursue and shoot down a target aircraft (20), according to one embodiment.

[0047] Referring to FIG. 1, in step S110, the aircraft (10) can detect a target aircraft (20) within an image captured by an image capturing device (1320). For example, the aircraft (10) can detect the target aircraft (20) within the image using a model for detecting objects.

[0048] The target aircraft (20) may be, for example, a drone (multiCopter), but the technical idea of ​​the present disclosure is not limited thereto, and the target aircraft (20) may be various unmanned aircraft such as VTOL (Vertical Takeoff and Landing) and Fixed-Wing.

[0049] In step S120, the aircraft (10) can obtain tracking information of the image capturing device (1320). When a target aircraft (20) is detected in the image, the aircraft (10) can control the direction and focus of the image capturing device (1320) based on the tracking information so that the target aircraft (20) can be continuously tracked. For example, the tracking information can include relative altitude information and relative direction information for the target aircraft (20) with respect to the aircraft (10).

[0050] In step S130, the aircraft (10) can obtain pose information of the target aircraft (20). For example, the aircraft (10) can extract feature points corresponding to the target aircraft (20) within the captured image, and based on information of the extracted feature points, can calculate a relative angle for estimating the movement direction of the target aircraft (20). Based on the relative angle, the aircraft (10) can obtain pose information of the target aircraft (20) including relative attitude information and movement direction information of the target aircraft (20).

[0051] In step S140, the aircraft (10) may generate a flight guidance command based on the mode and tracking information determined based on the relative angle. For example, the aircraft (10) may determine a response mode based on the range of the relative angle, and generate a flight guidance command based on the response mode and tracking information.

[0052] In step S150, the aircraft (10) can follow and shoot down the target aircraft (20) according to the flight guidance command.

[0053] In the above, for the convenience of explanation, the operation of one aircraft (10) following and shooting down a target aircraft (20) is described, but the technical idea of ​​the present disclosure is not limited thereto, and two or more aircraft (10) may follow and shoot down a target aircraft (20). When two or more aircraft (10) are used, each aircraft (10) maintains autonomy among itself and can share information in real time through a centralized or distributed control method, and can perform complex missions such as precise strikes on the target aircraft (20) and simultaneous attacks. When two or more aircraft (10) are used to perform a mission cooperatively, the two or more aircraft (10) may be referred to as a swarm drone.

[0054] Furthermore, the present invention can be used not only as a technology for shooting down target aircraft (20) from a defensive perspective, but can also be extended to an offensive perspective. For example, the present invention can be applied to detecting, shooting down, or attacking moving or stationary targets located on the ground or in the air using one or more aircraft (10) (including swarm drones). In other words, the present invention can be applied not only to military and security fields, but also in situations where targets must be effectively neutralized or eliminated.

[0055] Figure 2 is a flowchart showing an operation method of an aircraft (10) according to one embodiment.

[0056] The method of operating the aircraft (10) illustrated in FIG. 2 can be performed by at least one aircraft (10).

[0057] Referring to FIG. 2, in step S210, the aircraft (10) can detect a target aircraft (20) based on an image captured by an image capture device (1320). The image capture device (1320) can be mounted on the aircraft (10) and can capture images from various angles by adjusting the left-right direction, up-down direction, and left-right inclination using three axes: a horizontal (roll) axis, a vertical (pitch) axis, and a horizontal rotation (yaw) axis. For example, the target aircraft (20) can be an illegal aircraft (10) whose flight is not permitted. For example, the target aircraft (20) can be detected based on a detection model that detects an illegal aircraft (10) in an image. For example, the detection model can be a model learned through artificial intelligence.

[0058] In step S220, the aircraft (10) controls the image capturing device (1320) to track the target aircraft (20) based on the area of ​​the target aircraft (20) within the captured image and the center point within the captured image, and can obtain tracking information of the image capturing device (1320).

[0059] For example, the aircraft (10) can set a monitoring area including the area of ​​the target aircraft (20) within the captured image. A separate monitoring area can be set within the image to intensively track the target aircraft (20). For example, the monitoring area can be set in the form of a bounding box based on the boundary and size of the target aircraft (20). In order for the aircraft (10) to quickly and accurately track the illegal aircraft (10), the aircraft (10) can control the image capturing device (1320) so that the error between the center point of the monitoring area and the center point within the captured image is minimized. For example, the aircraft (10) can control the rotation of the three axes of the image capturing device (1320) so that the center point of the monitoring area and the center point within the captured image are aligned.

[0060] For example, the tracking information may be information acquired to detect the relative position and relative direction between an illegal aircraft (10) and an illegal aircraft (10) while the video recording device (1320) tracks the illegal aircraft (10). For example, the tracking information may include relative altitude information for a target aircraft (20) based on the aircraft (10) and relative direction information of the target aircraft (20) based on the aircraft (10). For example, the relative altitude information may be acquired based on the angle of the vertical (pitch) axis of the video recording device (1320). Additionally, the relative direction information may be acquired based on the angle of the horizontal rotation (yaw) axis of the video recording device (1320).

[0061] In step S230, the aircraft (10) can obtain pose information of the target aircraft (20) based on the captured image. Here, the pose information can indicate relative attitude information and movement direction information of the target aircraft (20) based on the aircraft (10) or the image capturing device (1320).

[0062] For example, the aircraft (10) can obtain coordinate information for preset locations of the target aircraft (20) based on a monitoring area including the area of ​​the target aircraft (20) within the captured image. For example, the aircraft (10) can obtain speed information including moving speed information and moving direction information of the target aircraft (20) based on the coordinate information with respect to the image capturing device (1320).

[0063] For example, the aircraft (10) may produce a first coordinate representing the upper left coordinate that is the farthest from the center coordinate of the monitoring area. The aircraft (10) may produce a second coordinate representing the upper right coordinate that is the farthest from the center coordinate of the monitoring area.

[0064] For example, the aircraft (10) can calculate a relative angle for estimating the movement direction of the target aircraft (20) based on the first coordinate and the second coordinate. The aircraft (10) can estimate the relative attitude information and movement direction information of the target aircraft (20) based on the sign of the relative angle with respect to the image capturing device (1320).

[0065] In step S240, the aircraft (10) can generate a flight guidance command for following and shooting down the target aircraft (20) based on the tracking information of the video recording device (1320) and the pose information of the target aircraft (20).

[0066] For example, the aircraft (10) can determine the predicted movement path of the target aircraft (20) and the impact point of the target aircraft (20) based on the tracking information of the video recording device (1320) and the pose information of the target aircraft (20). For example, the impact point may be a part that affects the flight stability of the target aircraft (20). Specifically, the impact point may be a propeller, a wing, or the like of the aircraft (10). The aircraft (10) can generate velocity vector information for tracking and shooting down the target aircraft (20) based on the predicted movement path and the impact point of the target aircraft (20). The aircraft (10) can follow the target aircraft (20) and shoot down the target aircraft (20) according to a flight guidance command corresponding to the velocity vector information.

[0067] For example, if a predetermined number of frames with the same relative angle sign are continuously detected from the captured image, the aircraft (10) can generate a flight guidance command based on a mode corresponding to the range of the relative angle. The process of calculating the relative angle is described in detail in FIGS. 8 and 9.

[0068] Specifically, if the absolute value of the relative angle satisfies the threshold value, the aircraft (10) can determine the first mode, which is the first state in which the target aircraft (20) does not move left and right. The aircraft (10) can generate first velocity vector information for tracking and shooting down the target aircraft (20) based on the tracking information of the video recording device (1320) according to the first mode.

[0069] On the other hand, if the absolute value of the relative angle exceeds the threshold value, the aircraft (10) can determine the second mode, which is a second state in which the target aircraft (20) moves left or right. The aircraft (10) can generate second velocity vector information for tracking and shooting down the target aircraft (20) based on the tracking information of the video recording device (1320) and the pose information of the target aircraft (20) according to the second mode.

[0070] FIG. 3 is a drawing for explaining a process of controlling an image capturing device (1320) to track a target aircraft (20) according to one embodiment.

[0071] Referring to FIG. 3, the aircraft (10) can capture an image near its movement path through an image capturing device (1320) within the aircraft (10). Based on the captured image, the aircraft (10) can detect a target aircraft (20). Referring to the image (310) of FIG. 3, the aircraft (10) can set a monitoring area (313) that includes the target aircraft (20) within the captured image. Here, the monitoring area (313) may be set by a user input or may be set by an artificial intelligence model that detects the target aircraft. The aircraft (10) can extract a center point (311) within the image. For example, the coordinate values ​​of the center point (311) within the image may be (c_u, c_v). In addition, the aircraft (10) can extract a center point (312) within the monitoring area (313). The coordinate values ​​of the center point (312) within the monitoring area (313) may be (t_u, t_v). The aircraft (10) may control the image capturing device (1320) so that the error between the center point (312) within the monitoring area (313) and the center point (311) within the image is minimized. Specifically, the error between the center point (311) and the center point (312) may be calculated as in mathematical equation 1.

[0072]

[0073] For example, the video capturing device (1320) can capture images using three axes: a horizontal (roll) axis, a vertical (pitch) axis, and a horizontal rotation (yaw) axis. By using three axes, the video capturing device (1320) can capture images stably even in situations where there is vibration or movement.

[0074] Referring to image (320) of FIG. 3, the aircraft (10) can control the image capturing device (1320) so that the center point (312) within the monitoring area (313) and the center point (311) within the image are aligned with each other.

[0075] Specifically, the aircraft (10) can calculate (G_pitch, G_yaw), which is a value of PWM (Pulse Width Modulation) for each axis based on PID (Proportional-Integral-Derivative), using the values ​​of (u_error, v_error). The (G_pitch, G_yaw) values ​​can be calculated according to mathematical expression 2. Here, mathematical expression 2 is an example of making the center point (312) within the monitoring area (313) and the center point (311) within the image coincide with each other through control of the vertical (pitch) axis and the horizontal rotation (yaw) axis.

[0076]

[0077] Here, K_p,pitch,K_i,pitch, K_d,pitch, K_p,yaw,K_i,yaw, K_d,yaw are tuning parameters.

[0078] The generated (G_pitch, G_yaw) values ​​are transmitted to the video recording device (1320), and the video recording device (1320) can operate to match the center point (312) within the monitoring area (313) and the center point (311) within the image using the (G_pitch, G_yaw) values.

[0079] Meanwhile, the center point (312) within the monitoring area (313) and the center point (311) within the image can be aligned with each other by controlling at least two axes among the horizontal (roll) axis, the vertical (pitch) axis, and the horizontal rotation (yaw) axis configured in the video recording device (1320). In this case, the mathematical expression used can be expressed differently from mathematical expression 2 depending on the type and number of axes used for control.

[0080] Although the process of controlling the video recording device (1320) to track a target aircraft (20) is described in FIG. 3, the technical concepts of the present disclosure are not limited thereto, and the process of tracking a target can be utilized for purposes other than national defense. For example, it can also be utilized for general object tracking, such as tracking people, objects, or moving vehicles.

[0081] In addition, the method for detecting and tracking a target aircraft (20) in the present disclosure is not limited to detecting and tracking a target based on RGB images, and can also be utilized for IR images. When IR images are utilized, the target aircraft (20) can be accurately detected and tracked even at night or in poor weather conditions. For example, a mode switch from RGB images to IR images can be performed at night or in cloudy weather. Fig. 4 is a diagram illustrating a coordinate system of an aircraft (10) according to one embodiment.

[0082] Referring to FIG. 4, the aircraft (10) can continuously track the target aircraft (20) and capture the target aircraft (20) through the image capturing device (1320). In this case, the aircraft (10) and the image capturing device (1320) can use different coordinate systems, and the coordinate system of the aircraft (10) and the coordinate system of the image capturing device (1320) are as follows.

[0083] The coordinate system of the aircraft (10) can be expressed as F_b=(x_b, y_b, z_b), the direction of the x_b axis can represent the heading direction of the aircraft (10), the direction of the y_b axis can represent the right direction of the aircraft (10), and the direction of the z_b axis can represent the downward direction of the aircraft (10).

[0084] In addition, the coordinate system of the image capturing device (1320) can be expressed as F_g=(x_g, y_g, z_g), and the direction of the x_g axis can indicate the right direction of the image capturing device (1320), the direction of the y_g axis can indicate the downward direction of the image capturing device (1320), and the direction of the z_g axis can indicate the front of the image capturing device (1320).

[0085] FIG. 5a is a drawing for explaining a process of obtaining tracking information of a target aircraft (20) for an aircraft (10) according to an embodiment, and FIG. 5b is a drawing for explaining a process of calculating velocity vector information of a target aircraft (20) using tracking information of the target aircraft (20) according to an embodiment.

[0086] Tracking information may include relative altitude information for a target aircraft (20) based on the aircraft (10) and relative direction information for the target aircraft (20) based on the aircraft (10).

[0087] Image (510) of Fig. 5a shows a side view of the aircraft (10) as viewed from the side, and image (520) of Fig. 5a shows a cross-sectional view as viewed from the top of the aircraft (10). Here, represents the angle from x_b to z_g', represents the angle from x_b to z_g''. can represent the angle of the vertical (pitch) axis of the video recording device (1320), can represent the angle of the horizontal rotation (yaw) axis of the video recording device (1320).

[0088] also, and Information on the target aircraft (20) can be obtained based on the angle information. Specifically, represents the relative altitude of the target aircraft (20) based on the aircraft (10), > 0, it may indicate that the target aircraft (20) is at a lower altitude than the aircraft (10). In addition, indicates the relative direction of the target aircraft (20) based on the aircraft (10), > If 0, it can indicate that the target aircraft (20) is located to the right of the heading of the aircraft (10).

[0089] For example, the aircraft (10) can generate a flight guidance command for following and shooting down a target aircraft (20) based on the tracking information of the video recording device (1320). In this case, the aircraft (10) can calculate the velocity vector information of the target aircraft (20) based on the tracking information.

[0090] For example, referring to the image (530) and image (540) of FIG. 5b, the aircraft (10) is pitched ( ), Yaw( ) Based on the angle information, a linear velocity vector can be generated according to mathematical expressions 3 and 4.

[0091]

[0092]

[0093] Here, v_max is the maximum speed of the aircraft (10).

[0094] Also, referring to the image (540) of FIG. 5b, the aircraft (10) has a Yaw ( ) Based on the angle information, an angular velocity vector can be generated according to mathematical expressions 5 and 6.

[0095]

[0096]

[0097] Here, , , is a tuning parameter.

[0098] FIG. 6 is a drawing for explaining a process in which an aircraft (10) performs a flight according to a flight guidance command and tracks a target aircraft (20), according to one embodiment.

[0099] The aircraft (10) can perform flight according to velocity vector information for tracking and shooting down a target aircraft (20). Fig. 6 shows an image in which the aircraft (10) tracks a target aircraft (20) according to a flight guidance command including velocity vector information. Specifically, the image (610) of Fig. 6 shows a side view of the aircraft (10) as viewed from the side, and the image (620) of Fig. 6 shows a cross-sectional view as viewed from the top of the aircraft (10).

[0100] For example, while following a target aircraft (20) according to a flight guidance command, the center point in the image captured by the image capturing device (1320) and the center point of the monitoring area including the image of the target aircraft (20) can be aligned.

[0101] FIG. 7 is a drawing for explaining a process of obtaining pose information of a target aircraft (20) according to one embodiment.

[0102]

[0103] The aircraft (10) can detect the target aircraft (20) from the image (701) captured by the image capturing device (1320). The aircraft (10) can set the area in the image (701) containing the target aircraft (20) as a monitoring area (721). In this case, in order for the aircraft (10) to continuously track the target aircraft (20), the operation of the image capturing device (1320) can be controlled so that the center point (711) within the monitoring area (721) and the center point of the image (701) are aligned with each other.

[0104] In step S710, the aircraft (10) may perform image binarization on the monitoring area (721) including the target aircraft (20). Binarization processing may be performed to extract the shape of the target aircraft (20) within the monitoring area (721), and a binarized image (702) may be generated. The image binarization process is described in detail in FIG. 8.

[0105] In step S720, the aircraft (10) can calculate coordinate information for feature points corresponding to preset locations of the target aircraft (20). For example, referring to image (703), a first coordinate indicating the upper left coordinate that is the farthest from the center coordinate of the monitoring area, and a second coordinate indicating the upper right coordinate that is the farthest from the center coordinate of the monitoring area can be calculated. The process of calculating coordinate information is described in detail in FIG. 9.

[0106] In step S730, referring to the image (704) and the image (705), the aircraft (10) can calculate a relative angle for estimating the movement direction of the target aircraft (20) based on the first coordinate and the second coordinate. The process of calculating the relative angle is described in FIG. 9. The aircraft (10) can generate a flight guidance command based on the tracking information and the pose information of the target aircraft (20). The process of generating the flight guidance command is described in detail in FIGS. 11 and 12.

[0107] FIG. 8 is a drawing for explaining a process of binarizing an image of a target aircraft (20) according to an embodiment.

[0108] Referring to FIG. 8, a monitoring area (801) including an area of ​​a target aircraft (20) may be set within an image (701) captured by an image capturing device (1320). Here, the monitoring area (801) may be set in the form of a bounding box based on the boundary and size of the target aircraft (20). In order to extract the shape of the target aircraft (20), binarization may be performed on an area within the upper left coordinates (u_1, v_1) and the lower right coordinates (u_2, v_2) of the monitoring area (801). The upper left coordinate may be calculated as the coordinate that is the farthest from the center coordinates (802) of the monitoring area (801), and the lower right coordinate may be calculated as the coordinate that is the farthest from the center coordinates (802) of the monitoring area (802).

[0109] A set of pixels representing the shape of a target aircraft (20) in binarized pixel values ​​can be defined as in mathematical expression 7, and an image (702) can be generated by performing binarization processing according to mathematical expression 8.

[0110]

[0111]

[0112] FIG. 9 is a drawing for explaining a process of calculating coordinates for feature points of a target aircraft (20) and calculating relative angles, according to one embodiment.

[0113] Referring to Fig. 9, the binarized image (702) can be divided into four parts based on the center coordinates (t_u, t_v). Here, the image processed image (702) may be an image of the monitoring area (802) described in Fig. 8. Among the C sets, the upper left area (901) and the upper right area (902) can be extracted according to mathematical expression 9.

[0114]

[0115]

[0116] In addition, the first coordinate (u_lt,v_lt) indicating the upper left coordinate (911) that is the farthest from the center coordinate (t_u,t_v) of the image (702) and the second coordinate (u_rt,v_rt) indicating the upper right coordinate (912) that is the farthest from the center coordinate (t_u,t_v) of the image (702) can be calculated according to mathematical expression 10.

[0117]

[0118]

[0119] Based on the first coordinate (911) and the second coordinate (912), a relative angle for estimating the movement direction of the target aircraft (20) with respect to the image capturing device (1320) can be calculated according to mathematical expression 11.

[0120]

[0121] FIG. 10 is a drawing for explaining a process of obtaining pose information of a target aircraft (20) based on a sign of a relative angle, according to one embodiment.

[0122] Based on the sign of the relative angle calculated according to mathematical expression 11 in FIG. 9, the relative attitude information and movement direction information of the target aircraft (20) can be estimated. For example, referring to image (1010) of FIG. 10, if the value of the relative angle is 0, the state of the target aircraft (20) can be determined as a state without left-right movement. For another example, referring to image (1020) of FIG. 10, if the sign of the relative angle is negative, the state of the target aircraft (20) can be determined as a state of moving to the left. For another example, referring to image (1030) of FIG. 10, if the sign of the relative angle is positive, the state of the target aircraft (20) can be determined as a state of moving to the right.

[0123] FIG. 11 is a drawing for explaining the results of comparing a case in which velocity vector information of a target aircraft (20) is obtained using tracking information of an image capture device (1320) and a case in which velocity vector information of a target aircraft (20) is obtained using tracking information of an image capture device (1320) and pose information of the target aircraft (20), according to one embodiment.

[0124] Image (1110) of Fig. 11 shows a cross-sectional view of the aircraft (10) as viewed from the top, as described in Fig. 5a. Here, represents the angle from x_b to z_g''. indicates the relative direction of the target aircraft (20) based on the aircraft (10), > If 0, it can indicate that the target aircraft (20) is located to the right of the heading of the aircraft (10).

[0125] Here, Since the next expected position of the target aircraft (20) is not taken into consideration, in order for the aircraft (10) to follow and hit the target aircraft (20), In addition to the tracking information of the video capturing device (1320) including information, the pose information of the target aircraft (20) may also be considered. Here, the pose information of the target aircraft (20) may include relative attitude information and movement direction information of the target aircraft (20) based on the video capturing device (1320). The relative attitude information and movement direction information may be determined based on a relative angle.

[0126] For example, if the sign of the relative angle is negative (1101), the next predicted position of the target aircraft (20) may be located on a path in the left direction. For another example, if the sign of the relative angle is positive (1102), the next predicted position of the target aircraft (20) may be located on a path in the right direction.

[0127] Image (1120) of Fig. 11 shows a cross-sectional view from the top of the aircraft (10) and shows a velocity vector determined based on tracking information and pose information of the target aircraft (20).

[0128] For example, if the sign of the relative angle is negative (1101), a velocity vector (1121) can be generated by considering the next expected position of the target aircraft (20) on a leftward path. For another example, if the sign of the relative angle is positive (1102), a velocity vector (1122) can be generated by considering the next expected position of the target aircraft (20) on a rightward path. The process of generating the velocity vector is described in FIG. 12. By generating the velocity vector by considering the next expected position of the target aircraft (20), the time for the aircraft (10) to shoot down the target aircraft (20), the hitting accuracy, and the shooting success rate can be increased.

[0129] FIG. 12 is a drawing for explaining a process of obtaining velocity vector information based on tracking information of an image capturing device (1320) and pose information of a target aircraft (20), according to one embodiment.

[0130] Referring to the image (1210) of FIG. 12, if the sign of the relative angle is positive (1102), the aircraft (10) can generate velocity vector information for tracking and shooting down the target aircraft (20) based on the tracking information of the video recording device (1320) and the pose information of the target aircraft (20).

[0131] Referring to the image (1220) of Fig. 12, the aircraft (10) is in Pitch ( ), Based on the angle information, a linear velocity vector can be generated according to Equations 12 and 13.

[0132]

[0133]

[0134] Here, v_max is the maximum speed of the aircraft (10), and K_angle is a preset gain value.

[0135] Additionally, the aircraft (10) Based on the angle information, an angular velocity vector can be generated according to Equations 14 and 15.

[0136]

[0137]

[0138] Here, , , is a tuning parameter.

[0139] Fig. 13 is a block diagram illustrating the configuration of an aircraft (10) according to one embodiment.

[0140] Referring to FIG. 13, the aircraft (10) may include a communication device (1310), an image capturing device (1320), a memory (1330), and a processor (1340). However, not all of the illustrated components are essential components. The aircraft (10) may be implemented with more components than the illustrated components, or with fewer components. The above components will now be described. The aircraft (10) illustrated in FIG. 13 may correspond to the aircraft (10) described in FIGS. 1 to 12 in the same manner.

[0141] The communication device (1310) can communicate with an external device. For example, the communication device (1310) can be connected to a network via a wired or wireless connection and communicate with the external device. Here, the external device may be an electronic device.

[0142] The communication device (1310) may include a communication module that supports one of various wired and wireless communication methods. The communication module may be a short-range communication module or a wired communication module.

[0143] The video recording device (1320) can be mounted on an aircraft (10) and can record videos from various angles by adjusting the left-right direction, up-down direction, and left-right tilt using three axes: a horizontal (roll) axis, a vertical (pitch) axis, and a horizontal rotation (yaw) axis. The video recording device (1320) can include a camera.

[0144] The video capture device (1320) can acquire image frames, such as still images or moving images, in a capture mode. Images captured by the video capture device (1320) can be processed by the processor (1340).

[0145] The memory (1330) can store at least one program for executing an operation method of the aircraft (10) that detects a target aircraft (20) through an image captured by the image capture device (1320) and generates a flight guidance command for following and shooting down the target aircraft (20) based on tracking information of the image capture device (1320) and pose information of the target aircraft (20). The at least one program stored in the memory (1330) can be classified into a plurality of modules according to function.

[0146] The processor (1340) controls the overall operation of the aircraft (10) and may include at least one processor (1340), such as a CPU. The processor (1340) may include at least one specialized processor (1340) corresponding to each function, or may be an integrated processor (1340).

[0147] The processor (1340) may execute a program stored in the memory (1330), read data or files stored in the memory (1330), or store new files in the memory (1330). In addition, the processor (1340) may execute instructions stored in the memory (1330).

[0148] The processor (1340) can detect a target aircraft (20) based on an image captured by the image capturing device (1320). For example, the target aircraft (20) may be an illegal aircraft (10) whose flight is not permitted. For example, the target aircraft (20) may be detected based on a detection model that detects an illegal aircraft (10) in the image. For example, the detection model may be a model learned through artificial intelligence.

[0149] The processor (1340) controls the image capturing device (1320) to track the target aircraft (20) based on the area of ​​the target aircraft (20) in the captured image and the center point in the captured image, and can obtain tracking information of the image capturing device (1320).

[0150] For example, the processor (1340) may set a monitoring area including the area of ​​the target aircraft (20) within the captured image. In order to intensively track the target aircraft (20), a separate monitoring area may be set within the image. For example, the monitoring area may be set in the form of a bounding box based on the boundary and size of the target aircraft (20). In order for the aircraft (10) to quickly and accurately track the illegal aircraft (10), the processor (1340) may control the image capturing device (1320) so that the error between the center point of the monitoring area and the center point within the captured image is minimized. For example, the processor (1340) may control the rotation of at least two of the three axes of the image capturing device (1320) so that the center point of the monitoring area and the center point within the captured image are aligned. For example, the video capturing device (1320) can control the rotation of the vertical (pitch) axis and the horizontal rotation (yaw) axis so that the center point of the monitoring area and the center point within the captured image are aligned.

[0151] For example, tracking information may be information acquired to detect the relative position and relative direction between an aircraft (10) and an illegal aircraft (10) while the video recording device (1320) tracks the illegal aircraft (10). For example, tracking information may include relative altitude information for a target aircraft (20) based on the aircraft (10) and relative direction information for the target aircraft (20) based on the aircraft (10).

[0152] The processor (1340) can obtain pose information of the target aircraft (20) based on the captured image. Here, the pose information can indicate relative attitude information and movement direction information of the target aircraft (20) based on the aircraft (10) or the image capturing device (1320).

[0153] For example, the processor (1340) may obtain coordinate information for preset locations of the target aircraft (20) based on a monitoring area including the area of ​​the target aircraft (20) within the captured image. For example, the processor (1340) may obtain speed information including movement speed information and movement direction information of the target aircraft (20) based on the coordinate information with respect to the image capturing device (1320).

[0154] For example, the processor (1340) may calculate a first coordinate representing the upper left coordinate that is the farthest from the center coordinate of the monitoring area. The processor (1340) may calculate a second coordinate representing the upper right coordinate that is the farthest from the center coordinate of the monitoring area.

[0155] For example, the processor (1340) can calculate a relative angle for estimating the movement direction of the target aircraft (20) based on the first coordinate and the second coordinate. The processor (1340) can estimate the relative attitude information and movement direction information of the target aircraft (20) based on the sign of the relative angle with respect to the image capturing device (1320).

[0156] The processor (1340) can generate a flight guidance command for following and shooting down the target aircraft (20) based on the tracking information of the video recording device (1320) and the pose information of the target aircraft (20).

[0157] For example, the processor (1340) can determine the predicted movement path of the target aircraft (20) and the impact point of the target aircraft (20) based on the tracking information of the video recording device (1320) and the pose information of the target aircraft (20). For example, the impact point may be a part that affects the flight stability of the target aircraft (20). Specifically, the impact point may be a propeller, a wing, etc. of the aircraft (10). The aircraft (10) can generate velocity vector information for tracking and shooting down the target aircraft (20) based on the predicted movement path and the impact point of the target aircraft (20). The processor (1340) can follow the target aircraft (20) and shoot down the target aircraft (20) according to a flight guidance command corresponding to the velocity vector information.

[0158] For example, if a preset number of frames with the same relative angle sign are continuously detected from the captured image, the processor (1340) may generate a flight guidance command based on a mode corresponding to the range of the relative angle.

[0159] Specifically, if the absolute value of the relative angle satisfies the threshold value, the processor (1340) may determine the target aircraft (20) to be in the first mode, which is a first state in which there is no left-right movement. The processor (1340) may generate first velocity vector information for tracking and shooting down the target aircraft (20) based on the tracking information of the video recording device (1320) according to the first mode.

[0160] On the other hand, if the absolute value of the relative angle exceeds the threshold value, the processor (1340) may determine the second mode, which is a second state in which the target aircraft (20) moves left or right. The processor (1340) may generate second velocity vector information for tracking and shooting down the target aircraft (20) based on the tracking information of the video capturing device (1320) and the pose information of the target aircraft (20) according to the second mode.

[0161] The aircraft (10) described in the present disclosure may be implemented using hardware components, software components, and / or a combination of hardware components and software components. Furthermore, the present disclosure may be provided in the form of a computer program stored on a computer-readable storage medium to perform the operating method of the aircraft (10). Furthermore, the present disclosure may be written as a computer-executable program and implemented on a general-purpose digital computer that executes such a program using a computer-readable storage medium.

[0162] Such computer-readable storage media may be read-only memory (ROM), random-access memory (RAM), flash memory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, magnetic tape, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state disks (SSDs), and any device capable of storing instructions or software, related data, data files, and data structures, and providing instructions or software, related data, data files, and data structures to a processor or a computer so that the processor or the computer may execute the instructions.

[0163] Although the embodiments have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of the present invention defined in the following claims also fall within the scope of the present invention.

Claims

1. A method of operating an aircraft performed by at least one aircraft, A step of detecting a target aircraft based on an image captured by an image capturing device capable of capturing RGB images or IR images; A step of controlling the image capturing device to track the target aircraft based on the area of ​​the target aircraft within the captured image and the center point within the captured image, and obtaining tracking information of the image capturing device; A step of obtaining pose information of the target aircraft based on the captured image; and A method for operating an aircraft, comprising the step of generating a flight guidance command for following and shooting down the target aircraft based on tracking information of the video recording device and pose information of the target aircraft.

2. In paragraph 1, The step of controlling the above video recording device is: A step of setting a monitoring area including the area of ​​the target aircraft within the captured image; and A method of operating an aircraft, comprising the step of controlling the image capturing device so that an error between a center point of the monitoring area and a center point within the captured image is minimized.

3. In paragraph 1, The above tracking information is, It includes relative altitude information for the target aircraft with respect to the at least one aircraft and relative direction information of the target aircraft with respect to the at least one aircraft, The above target aircraft corresponds to at least one of MultiCopter, VTOL (Vertical Takeoff and Landing), and Fixed-Wing. How the aircraft operates.

4. In paragraph 1, The step of obtaining pose information of the above target aircraft is: A step of obtaining coordinate information for preset locations of the target aircraft based on a monitoring area including an area of ​​the target aircraft within the captured image; and A method for operating an aircraft, comprising: obtaining speed information including moving speed information and moving direction information of the target aircraft based on the coordinate information, with respect to the image capturing device.

5. In paragraph 4, The step of obtaining the above coordinate information is: A step of calculating a first coordinate representing the upper left coordinate that is the farthest from the center coordinate of the above monitoring area; and A method of operating an aircraft, comprising the step of calculating a second coordinate representing the upper right coordinate furthest from the center coordinate of the monitoring area.

6. In paragraph 5, The step of obtaining the above pose information is: A step of calculating a relative angle for estimating the movement direction of the target aircraft based on the first coordinate and the second coordinate; and A method for operating an aircraft, comprising the step of estimating relative attitude information and movement direction information of the target aircraft with respect to the image capturing device based on the sign of the relative angle.

7. In paragraph 1, The step of generating the above flight guidance command is: A step of determining a predicted movement path of the target aircraft and a striking point of the target aircraft based on the tracking information of the video capturing device and the pose information of the target aircraft; and A method for operating an aircraft, comprising the step of generating velocity vector information for tracking and shooting down the target aircraft based on the predicted movement path of the target aircraft and the impact point.

8. In paragraph 1, The step of generating the above flight guidance command is: A method for operating an aircraft, comprising the step of generating the flight guidance command based on a mode corresponding to the range of the relative angle when frames having the same sign of the relative angle are continuously detected a preset number or more from the captured image.

9. In paragraph 8, The step of generating the above flight guidance command is: If the absolute value of the relative angle satisfies the threshold value, the step of determining the target aircraft as a first mode, which is a first state in which there is no left-right movement; and A method for operating an aircraft, comprising: generating first velocity vector information for tracking and shooting down a target aircraft based on tracking information of the video recording device according to the first mode.

10. In paragraph 8, The step of generating the above flight guidance command is: If the absolute value of the relative angle exceeds the threshold value, the step of determining the target aircraft as a second mode, which is a second state in which the target aircraft moves left or right; and A method for operating an aircraft, comprising: generating second velocity vector information for tracking and shooting down the target aircraft based on tracking information of the video recording device and pose information of the target aircraft according to the second mode.

11. Communication devices; video recording device; processor; and comprising a memory storing instructions executable by the processor; The above processor, by executing the above instructions, Based on the image captured by the above image capturing device, the target aircraft is detected, Controlling the image capturing device to track the target aircraft based on the area of ​​the target aircraft within the captured image and the center point within the captured image, and obtaining tracking information of the image capturing device; Based on the captured image, pose information of the target aircraft is obtained, An aircraft that generates a flight guidance command for following and shooting down the target aircraft based on the tracking information of the video recording device and the pose information of the target aircraft.

12. In paragraph 11, The above processor, by executing the above instructions, A monitoring area including the area of ​​the target aircraft is set within the above-described captured image, An aircraft that controls the image capturing device so that the error between the center point of the monitoring area and the center point within the captured image is minimized.

13. In paragraph 11, The above processor, by executing the above instructions, Based on the monitoring area including the area of ​​the target aircraft within the captured image, coordinate information for preset locations of the target aircraft is obtained, An aircraft that obtains speed information including moving speed information and moving direction information of the target aircraft based on the above coordinate information and relative to the image capturing device.

14. In paragraph 13, The above processor, by executing the above instructions, A first coordinate is calculated, which represents the upper left coordinate that is furthest from the center coordinate of the above monitoring area, An aircraft that produces a second coordinate representing the upper right coordinate furthest from the center coordinate of the above monitoring area.

15. In paragraph 14, The above processor, by executing the above instructions, Based on the first coordinate and the second coordinate, a relative angle for estimating the movement direction of the target aircraft is calculated, An aircraft that estimates relative attitude information and movement direction information of the target aircraft based on the sign of the relative angle with respect to the image capturing device.

16. In paragraph 11, The above processor, by executing the above instructions, An aircraft that generates a flight guidance command based on a mode corresponding to the range of the relative angle when frames having the same sign of the relative angle are continuously detected a preset number or more from the captured image.

17. In paragraph 16, The above processor, by executing the above instructions, If the absolute value of the above relative angle satisfies the threshold value, the target aircraft is determined to be in the first mode, which is the first state in which there is no left-right movement. An aircraft that generates first velocity vector information for tracking and shooting down the target aircraft based on tracking information of the video recording device according to the first mode.

18. In paragraph 16, The above processor, by executing the above instructions, If the absolute value of the above relative angle exceeds the threshold value, the target aircraft is determined to be in the second mode, which is a second state in which the target aircraft moves left or right, An aircraft that generates second velocity vector information for tracking and shooting down the target aircraft based on tracking information of the video recording device and pose information of the target aircraft according to the second mode.

19. In paragraph 11, The above target aircraft is an aircraft corresponding to a moving or fixed target located on the ground or in the sky.

20. A computer-readable, non-transitory recording medium having recorded thereon a program for executing the method of operating an aircraft described in Article 1.

Images