Drone system, drone control program, and drone control method

Abstract

A drone system 100 comprises: a main drone 10 including a first imaging means 11; a sub drone 20 including a second imaging means 21 for imaging the main drone 10 from a bird's eye view; and a controller 30 for controlling the main drone 10 and the sub drone 20. The controller 30 includes a display section 35 for displaying a bird's-eye-view image captured by the second imaging means 21, and has a function of displaying a first marker indicating, in the bird's-eye-view image, a distance from the main drone.

Description

Technical Field

[0001] The technology described in this specification relates to a drone system, a drone control program, and a drone control method.

Background Art

[0002] In recent years, drones have been used in various applications, such as aerial photography, inspection, pesticide spraying, disaster relief, and cargo transportation, and the market size of the drone business has been continuously expanding. Manual operation of the drone is required in situations where the intention of the operator is important, but the difficulty of operation is very high. In particular, in "out-of-sight flight" where a drone outside the operator's field of vision is operated relying only on the camera image of the aircraft, the operator's understanding of the surrounding situation (Situational Awareness: SA) becomes poor. The factors that cause the operator's SA to deteriorate are that the viewing angle of the camera of the aircraft is narrow with respect to the degrees of freedom of movement of the drone, and it is difficult for the operator to grasp the height of the drone and the distance from the target object only from the camera image. That is, it is difficult to fly the drone safely at a remote location only with the information of the camera image of the aircraft.

[0003] SA is classified into level 1, a state where the state around the aircraft can be perceived; level 2, a state where the state around the aircraft can be understood; and level 3, a state where the state around the aircraft can be predicted. As the level increases, the surrounding situation can be understood, and the tasks performed by the drone can be effectively improved. However, current drones judge from a first-person perspective only based on the images mounted on the drone, and there are many blind spots that are not captured by the camera, and often do not even meet level 1.

[0004] Technologies are known that improve SA (Sight Assurance) and support drone operation. Patent Document 1 discloses a method for displaying multiple images transmitted from multiple drones on a user's display device, and for displaying images of other drones from an overhead perspective depending on the selection of images captured by the multiple drones. Patent Document 2 discloses an overhead view display system that converts and displays an image of a mobile device equipped with multiple first cameras that capture the surroundings in a wide angle and a second camera that captures the direction of travel, from a virtual overhead viewpoint. Patent Document 3 discloses an image display method that displays an image that appears to be an overhead view including the drone, using an imaging device mounted on the drone.

[0005] Patent documents 1 to 3 describe how a virtual, or in some cases, overhead view images can be obtained, but it is not possible to constantly monitor the drone's status in real time. The inventors of this application have proposed a technology that allows for constant, real-time monitoring of the main drone's status and improves drone operability by capturing an overhead image of the main drone's status using a spatially linked sub-drone to obtain a third-person perspective (for example, Non-Patent Document 1). [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2019-195176 [Patent Document 2] Japanese Patent Publication No. 2020-161895 [Patent Document 3] U.S. Patent Publication No. 2019 / 373184 [Non-patent literature]

[0007] [Non-Patent Document 1] Ryotaro Tenma et al., "Extension of Drone Control Interface Using Two Spatially Interlocking Camera Viewpoints," Transactions of the Information Processing Society of Japan (Web), Vol. 61 No. 8, pp. 1319-1332. [Overview of the project] [Problems that the invention aims to solve]

[0008] However, the technology proposed by the inventors of the present invention, which merely involves spatial coordination between the main drone and the sub-drone, has limitations. While it makes it easier for the operator to check blind spots around the drone, it does not allow for accurate perception of the space around the main drone, specifically the distance from the main drone. Therefore, when attempting to have a drone perform tasks such as pesticide spraying or delivery in situations that do not meet SA Level 2 standards, significant problems arise, such as inability to spray in the correct location, dropping delivered items far from the target, or colliding with the target. To safely operate a drone remotely, it is necessary for the operator to be able to check blind spots and accurately perceive the space around the drone.

[0009] In one respect, the technology described herein aims to provide a highly visible image display that allows the operator to check blind spots around the drone and accurately perceive the space surrounding the drone. [Means for solving the problem]

[0010] In one aspect, the drone system comprises a main drone equipped with a first imaging means, a sub-drone equipped with a second imaging means for imaging the main drone from an overhead perspective, and a controller for controlling the main drone and the sub-drone, wherein the controller is Area image captured by the first imaging means and The controller may also include a display unit that displays the overhead view image captured by the second imaging means. A first marker is superimposed on the image to determine the distance from the main drone. It may have a display function. [Effects of the Invention]

[0011] One aspect of this system is to provide a highly visible image display that allows the operator to check blind spots around the drone and accurately perceive the surrounding space. [Brief explanation of the drawing]

[0012] [Figure 1] This is a block diagram schematically showing an example configuration of a drone system as an embodiment. [Figure 2] This is an example of images captured by the main drone and the secondary drone at approximately the same time. [Figure 3] This diagram illustrates the positions of the main drone and secondary drone using real-world coordinates, pixel coordinates, and third-person view coordinates. [Figure 4] This diagram illustrates the distance between the main drone and the secondary drone. [Figure 5] This figure shows markers displayed on the top-perspective view (TPV) image shown in Figure 2. [Figure 6] This figure shows an example of a display device showing both an area image and an overhead view image. [Figure 7] This flowchart shows an example of the control performed by the controller. [Figure 8] This diagram shows the relative positions of the primary drone and the secondary drone when there is an obstacle between them. [Modes for carrying out the invention]

[0013] The embodiments will now be described with reference to the drawings. However, the embodiments shown below are merely illustrative, and there is no intention to exclude various modifications or applications of techniques not explicitly shown in the embodiments. In other words, these embodiments can be implemented with various modifications without departing from their spirit.

[0014] Furthermore, each figure is not intended to represent only the components shown in the figure, but may include other components. In the following figures, parts denoted by the same reference numerals indicate the same or similar parts unless otherwise specified.

[0015] [1. Overall Structure] Figure 1 is a schematic block diagram showing an example configuration of a drone system 100 as an embodiment. The drone system 100 comprises a main drone 10, a sub-drone 20, and a controller 30. As shown in Figure 1, the two drones 10 and 20 and the controller 30 are wirelessly connected to each other via a network such as the Internet or Wi-Fi Direct.

[0016] The main drone 10 is the aircraft controlled by the user and is used for imaging and various tasks. The secondary drone 20 flies automatically and autonomously and is set to follow the main drone 10 at a higher position and a certain distance away. The main drone 10 and the secondary drone 20 may be equipped with devices that detect and automatically avoid obstacles. Here, obstacles include anything that hinders drone flight, whether natural or artificial.

[0017] [1-1. Main Drone Configuration] The main drone 10 is equipped with a camera 11, a sensor 12, a communication device 13, and a controller 14.

[0018] Camera 11 captures images of the surrounding environment (area) of the main drone 10. The images of the surrounding environment captured by the main drone 10 may be called area images. The orientation and tilt (angle in the focal field of view) of camera 11 are arbitrary. In this embodiment, the orientation of camera 11 is approximately the same as the direction of travel of the main drone 10, and the tilt of camera 11 is automatically or manually set to an angle suitable for capturing images of the area in front of the aircraft. Camera 11 is an example of a first imaging means.

[0019] Sensor 12 measures the three-dimensional position (latitude, longitude, altitude, orientation) and speed of the main drone 10.

[0020] The communicator 13 transmits to the controller 30 the area image (area information) captured by the camera 11, the orientation of the main drone 10 (direction of area image (first-person view, FPV) shooting), altitude, and speed measured by the sensor 12. The communicator 13 may also transmit the orientation and tilt of the camera 11 to the controller 30. The communicator 13 may also transmit signals to the sub-drone 20. The communicator 13 is an example of a communication means for the main drone 10.

[0021] The camera 11, sensor 12, and communication device 13 may be movable and equipped with a drive mechanism, allowing their angles to be freely changed.

[0022] The controller 14 controls the camera 11, sensor 12, and communication device 13, and controls the flight mechanism (not shown) of the main drone 10 and the drive mechanisms (not shown) of each component 11, 12, and 13 in accordance with the signals received by the communication device 13 from the controller 30.

[0023] [1-2. Configuration of the secondary drone] The secondary drone 20 is equipped with a camera 21, a sensor 22, a communication device 23, and a controller 24.

[0024] Camera 21 captures images of the main drone 10 from an overhead perspective. The orientation and tilt of camera 21 are arbitrary. In this embodiment, the orientation of camera 21 is approximately the same as the direction of travel of the secondary drone 20, and the tilt of camera 21 is automatically or manually set to an angle suitable for capturing images of the surrounding environment including the main drone 10. Camera 21 is an example of a second imaging means.

[0025] Sensor 22 measures the three-dimensional position (latitude, longitude, altitude, orientation) and speed of the secondary drone 20.

[0026] The communication device 23 transmits to the controller 30 the image (overhead view image) taken by the camera 21 of the main drone 10 from an overhead perspective, the tilt of the camera 21, and the orientation and altitude of the secondary drone 20 measured by the sensor 22. The communication device 13 may also transmit the orientation of the camera 21 and the speed of the aircraft to the controller 30. The communication device 23 may also transmit a signal to the main drone 10. The communication device 23 is an example of a communication means for the secondary drone 20.

[0027] The camera 21, sensor 22, and communication device 23 may be movable and equipped with a drive mechanism, allowing their angles to be freely changed.

[0028] The controller 24 controls the camera 21, sensor 22, and communication device 23, and controls the flight mechanism (not shown) of the sub-drone 20 and the drive mechanisms (not shown) of each component 21, 22, and 23 according to signals received by the communication device 23 from the controller 30.

[0029] [1-3. Controller Configuration] The controller 30 comprises a CPU 31, memory 32, storage device 33, IF unit 34, and display device 35. The controller 30 may be implemented as a single unit, or it may be implemented as a combination of separate devices such as a general-purpose computer, display device, and operating terminal.

[0030] The CPU 31 is a processing unit that performs various control and calculations, and realizes various functions by executing the Operating System (OS) and programs (drone control programs) stored in the memory 32. That is, as shown in Figure 1, the CPU 31 may function as an optimization unit 31a, a position calculation unit 31b, and a marker generation unit 31c.

[0031] The CPU 31 is an example of a computer and, exemplarily, controls the operation of the entire drone system 100. The device for controlling the operation of the entire drone system 100 is not limited to the CPU 31, and may be, for example, one of the following: MPU, DSP, ASIC, PLD, FPGA, or dedicated processor. Alternatively, the device for controlling the operation of the entire drone system 100 may be a combination of two or more of the following: CPU, MPU, DSP, ASIC, PLD, FPGA, and dedicated processor. Note that MPU is an abbreviation for Micro Processing Unit, DSP is an abbreviation for Digital Signal Processor, and ASIC is an abbreviation for Application Specific Integrated Circuit. Also, PLD is an abbreviation for Programmable Logic Device, and FPGA is an abbreviation for Field Programmable Gate Array.

[0032] Memory 32 is a device that stores various data and information such as programs. Examples include volatile memory such as Dynamic Random Access Memory (DRAM) and non-volatile memory such as Persistent Memory (PM), or both.

[0033] The storage device 33 is a device that stores data in a read-write manner, and may be, for example, a Hard Disk Drive (HDD), Solid State Drive (SSD), or Storage Class Memory (SCM). The storage device 33 stores various information received from the main drone 10 and the sub-drone 20, as well as values ​​calculated by the CPU 31.

[0034] The storage device 33 may store a program 36 (drone control program) that implements all or part of the various functions of the drone system 100. For example, the CPU 31 of the controller 30 can implement various functions of the controller 30 by loading the program 36 stored in the storage device 33 into the memory 32 and executing it.

[0035] The IF unit 34 is an example of a communication interface that controls the connection and communication between the controller 30 and the main drone 10, the sub-drone 20, and a network (not shown). In other words, the IF unit 34 may function as a communication means for the controller 30. For example, the IF unit 34 may include an adapter compliant with a Local Area Network (LAN) such as Ethernet®, or optical communication such as Fibre Channel (FC). The adapter may support wireless, wired, or both communication methods.

[0036] For example, the controller 30 may be connected to external devices or a cloud server (not shown) via the IF unit 34 and a network to enable communication with each other. The program used to control the drone system 100, including program 36, may be downloaded from the network to the controller 30 via the communication IF and stored in the storage device 33.

[0037] The display device 35 presents images or videos output from the controller 30 to the user. In other words, the display device 35 may function as a display unit. The display device 35 may be a touch panel and may accept input from the user. User input may be received by various input devices not shown. The display device 35 may display at least an area image received from the main drone 10 and an overhead view image received from the sub-drone 20 on two screens, and may also display markers indicating information about the space surrounding the main drone 10 within the displayed overhead view image.

[0038] [2. Captured Images] Figure 2 shows an example of images captured by the main drone 10 and the secondary drone 20 at approximately the same time. The two frames in Figure 2 represent the imaging ranges of the main drone 10 and the secondary drone 20, respectively.

[0039] The image on the right side of the imaging range of the main drone 10 is an area image (FPV) captured by the main drone 10. FPV stands for First Person View, meaning it is an image from the main drone's perspective. In this case, the camera 11 of the main drone 10 is imaging a certain range in the area in front of the main drone 10 in the direction of its movement.

[0040] The image on the right side of the imaging range of the secondary drone 20 is an overhead view image (TPV) captured by the secondary drone 20. TPV stands for Third Person View, meaning it is an image from an overhead view of the main drone 10. The overhead view image includes the main drone 10 approximately in the center, as well as the environment surrounding the main drone 10.

[0041] In this way, the overhead view image visualizes the area around the main drone 10, allowing the operator to check blind spots in real time. However, these two types of images alone are not sufficient to recognize the space around the main drone 10.

[0042] To provide a highly visible image display for accurately perceiving the surrounding space of the main drone 10, it is desirable that the secondary drone 20 always obtains information that allows it to understand and predict the surrounding space, depending on the speed of the main drone 10. To achieve this, the position of the secondary drone 20 needs to be optimized. In particular, since the operator controls the main drone 10 while viewing two screens on the display device 35, if the difference in distance between the main drone 10 and the secondary drone 20 becomes large, the two images displayed on the display device 35 will appear discontinuous. This may increase the operator's burden and reduce the quality of control. To prevent this, the distance traveled by the secondary drone 20 needs to be minimized. Furthermore, if the positional relationship between the main drone 10 and the sub-drone 20 is not appropriate, it becomes very difficult to accurately perceive the space surrounding the main drone 10. By fixing the positional relationship between the main drone 10 and the sub-drone 20 within a predetermined range, the distance from the main drone 10, as well as the imaging area or direction of travel of the main drone 10, can be displayed on the screen of the display device 35 at any time. Furthermore, the relationship between the obtained overhead view image and the area image (which region within the overhead view image the main drone 10 is capturing as an area image) becomes clearer, making the relationship between the overhead view image and the area image even more apparent. These controls are implemented by the functions of controller 30.

[0043] [3. Controller Functional Configuration] As shown in Figure 1, the CPU 31 of the controller 30 includes, as a functional configuration, an optimization unit 31a, a position calculation unit 31b, a marker generation unit 31c, and a display unit 35 in order to realize the drone control method of this embodiment. The optimization unit 31a optimizes the placement of the sub-drone 20 relative to the main drone 10. The position calculation unit 31b acquires the relative positions of the main drone 10 and the sub-drone 20 at the optimized position. The marker generation unit 31c visualizes information that enables understanding and prediction of the surrounding space of the main drone 10 based on the positional relationship between the main drone 10 and the sub-drone 20. Each function will be described below.

[0044] [3-1. Calculation of the optimal position by the position control unit]

[0045] The optimization unit 31a optimizes the placement of the sub-drone 20 relative to the main drone 10, specifically controlling the position of the sub-drone 20, the angle of the camera 21, and the target relative angle of the sub-drone 20. The optimization unit 31a is an example of a control unit. The optimization unit 31a is primarily based on automatic control, but may be switchable to manual control depending on the situation.

[0046] In order for the secondary drone 20 to capture an overhead view image of an appropriate range according to the speed of the main drone 10, it is important that the following conditions (a) to (c) are met. <Optimal positioning conditions> (a) The main drone 10 is positioned within the overhead view of the secondary drone 20. (i) The area captured by the camera 11 of the main drone 10 is within the overhead view of the secondary drone 20. (c) The area in which the main drone 10 moves within a predetermined time is included. The following describes a method for positioning the secondary drone 20 in a location that satisfies conditions (a) to (c).

[0047] <Optimization Methods> The placement of the secondary drone 20 is determined using an optimization method. The optimal position is, Step 1: Define the variables to optimize. Step 2: Define the cost function Step 3: Set constraints on variables Step 4: Output the target coordinates using the principal-dual interior point method. It is calculated by performing steps 1 and 2. The objective function obtained by performing steps 1 and 2 is shown in Equation 1.

number

[0048] Here, P(t): the 3D target relative coordinate vector of the secondary drone at time t, P init : Initial (before movement) 3D target relative coordinate vector of the secondary drone, θ f (t): Target camera angle of the secondary drone at time t, θ finit : Initial (before movement) target camera angle of the secondary drone, φ f (t): The relative angle of the secondary drone to the target at time t, and w 1~4 : This is the weighting coefficient. In other words, φ f (t) is the target angle of the secondary drone 20 relative to the main drone 10.

[0049] The (1) enclosed by the square in Equation 1 is referred to as the first term, (2) as the second term, and (3) as the third term. The first term is introduced for the purpose of minimizing the movement of the sub-drone 20 (minimizing the movement of the operator's perspective), the second term is introduced for the purpose of making the orientations of the main drone 10 and the sub-drone 20 coincide as much as possible, and the third term is introduced to return to the positional relationship specified by the operator at the end of the movement. In other words, it is introduced to control so as not to deviate much from the initial position.

[0050] Furthermore, set the constraint conditions of the objective function in Equation 1 (Step 3). The constraint conditions are the conditions shown in the following i to vii. The constraint conditions of i to vii are set for suppressing the angular difference between the perspective and the controlled object, the movable range of the camera gimbal of the sub-drone 20, the presentation range of the field of view in each forward direction, the display range of the frame indicating the shooting range of the main drone 10, etc.

Number

[0051] Here, as described above, φ f (t) is the target relative angle of the sub-drone 20 at time t (the target angle of the sub-drone 20 with respect to the main drone 10), and θ f (t) is the target camera angle of the sub-drone 20 at time t. Furthermore, x f (t), y f (t), z f (t) are the target relative coordinate vectors of each axis of the sub-drone 20 at time t, and θ fvfov is the vertical viewing angle of the camera 21 of the sub-drone 20, and θ fhfov is the horizontal viewing angle of the camera 21 of the sub-drone 20. Also, v x (t), v z (t) are the speeds of each axis of the main drone 10 at time t, y(t) is the altitude of the main drone 10 from the ground surface at time t, and θ mvfov indicates the vertical viewing angle of the camera of the main drone 10.

[0052] In FIG. 3, three-dimensional actual coordinates for explaining the positions of the main drone 10 and the sub-drone 20 are shown. The above xf (t), y f (t), z f (t), φ f (t) and θ f (t) represents the parameter (value) in the actual coordinates. Also, the dashed lines extending from each drone in the actual coordinates in Figure 3 indicate the direction of travel of each drone.

[0053] Finally, under various constraints, the optimization problem is solved using the objective function of Equation 1. The optimal algorithm for solving this optimization problem is, for example, the principal-dual interior-point method. The optimization unit 31a takes into account the current three-dimensional positions of the principal drone 10 and the sub-drone 20, and the angle θ of the camera 21 of the sub-drone 20. f (t), target relative angle φ f By using the primary-dual interior point method with (t), etc., the optimal placement position (three-dimensional target position) of the secondary drone 20, the angle of the camera 21, and the target relative angle of the secondary drone 20 are calculated.

[0054] The optimization unit 31a calculates control variables using cascaded PID and adjusts the position of the sub-drone 20, the angle of the camera 21, and the target relative angle of the sub-drone 20 based on the control variables. The optimization unit 31a controls the sub-drone 20 to move to the determined three-dimensional target position. Furthermore, it updates the angle of the camera 21 and the target relative angle of the sub-drone 20 at that position. In other words, the optimization unit 31a transmits a signal via the IF unit 34 to the communication device 23 of the sub-drone 20 to control the main drone 10 so that it is within the overhead view captured by the camera 21 of the sub-drone 20.

[0055] The optimization unit 31a may control the orientation of the sub-drone 20 according to the direction of movement of the main drone 10. For example, if the main drone 10 starts moving to the right, the optimization unit 31a rotates the orientation of the sub-drone 20 (or the orientation of the camera 21) so that the surrounding environment to the right of the main drone 10 is captured more broadly than the surrounding environment to the left. This allows for imaging of a wide area ahead of the direction of movement of the main drone 10.

[0056] As described above, with the control of the optimization unit 31a, the operator only needs to control the main drone 10, and the sub-drone 20 will follow the main drone 10 so that it stays within a predetermined range, and provide an overhead view image on the display device 35 screen. Therefore, the operator can concentrate on operating and working with the main drone 10 while also referring to the overhead view image of the sub-drone 20 obtained by automatic control, eliminating the need for complex operations and greatly improving operability. The optimization unit 31a may switch the automatic tracking control of the secondary drone 20 relative to the main drone to manual control depending on the situation. Here, tracking control means controlling the secondary drone to fly in a position where the main drone 10 is within a predetermined range and can be seen from above. At this time, when manually controlling the secondary drone 20, the control by the optimization unit 31a described above may be applied to the main drone 10 so that the main drone 10 automatically leads the secondary drone 20 so that it comes into the secondary drone 20's overhead field of view. Furthermore, a function may be provided to disable automatic tracking of the main drone 10 and the sub-drone 20, allowing both to be controlled manually.

[0057] The optimization unit 31a arranges the sub-drone 20 and the main drone 10 within a predetermined range using the method described above. Furthermore, after calculating the coordinate positions described later, the optimization unit 31a calculates the optimal position for the sub-drone 20 and places the sub-drone 20 at that position, thereby enabling the main drone 10 and the sub-drone 20 to be positioned in an accurate relative position. Furthermore, the optimization unit 31a calculates the optimal angle and the optimal relative angle of the camera 21 of the sub-drone 20 at the optimized position, and controls them accordingly, thereby enabling the main drone 10 to capture images at the optimal position and angle.

[0058] [3-2. Calculation of relative position by the position calculation unit] The optimization unit 31a ensures that the positional relationship between the main drone 10 and the sub-drone 20, the angle of the camera 21, and the relative angle of the sub-drone 20 are within an appropriate range, thereby enabling the main drone 10 to understand the surrounding space. In order to obtain information that enables understanding and prediction of the space surrounding the aircraft, it is first necessary to obtain the relative positions of the main drone 10 and the sub-drone 20. The position calculation unit 31b calculates the relative positions of the main drone 10 and the sub-drone 20.

[0059] While there are methods that use GPS to determine the relative positions of the main drone 10 and the secondary drone 20, the coordinates obtained by GPS have a long update frequency and a large error of 1-2m or more, making them unsuitable for drone control. Therefore, this invention employs a method that calculates the relative positions using image processing.

[0060] First, the position calculation unit 31b detects the main drone 10 from within the overhead view image. The detection method may be a well-known image processing technique, for example, using OpenCV to identify the main drone 10 by color. Furthermore, the position calculation unit 31b corrects the overhead view image. Image correction may be performed using a well-known method, for example, using a camera matrix and distortion matrix acquired in advance using Zhang's image recognition method.

[0061] Figure 3 shows the pixel coordinates and third-person viewpoint coordinates to describe the positions of the main drone 10 and the sub-drone 20. Based on the corrected image, the position calculation unit 31b calculates the position of the main drone 10 as seen by the camera 21 of the sub-drone 20 (coordinates x in real space (real world)). f (t), z f (t) is the pixel coordinate x on the horizontal plane of the main drone. pixel ,y pixel Next, the position calculation unit 31b calculates the pixel coordinate x of the main drone shown in the image. pixel ,y pixel Using the distance, the coordinates x in the third-person view are set with the image center M of the main drone 10 as the origin. m (t), y m (t) is calculated. Furthermore, the coordinates x in real space (real world) are obtained from the coordinates within the third-person view.f (t), z f The distance is calculated by converting it to (t). As shown in Figure 3, the coordinates of the main drone's position can be expressed in real coordinates, third-person view coordinates, and pixel coordinates.

[0062] <Distance x in third-person view coordinates> m ,y m Derivation of > coordinate x m ,y m In deriving this, the spatial plane in which the main drone 10 and the sub-drone 20 exist is represented in a coordinate system. Figure 4 is a diagram illustrating the distance between the main drone 10 and the sub-drone 20. The horizontal plane in which the main drone 10 exists and the vertical field of view angle θ of the camera 21 of the sub-drone 20. fvfov Let S and R be the intersection points where the line segments forming the triangle intersect, and let A be the position of camera 21 (for example, the lens position). Then, determine the line segments AS and AR. When comparing line segments AS and AR, take point E on the longer line segment AS such that the distance from point A is the same as that of the shorter line segment AR. At this time, an isosceles triangle AER is formed. Consider the perpendicular from camera 21's position A to isosceles triangle AER, and let M be the intersection point of the perpendicular from A to isosceles triangle AER, and let C be the intersection point of the extension of AM and the horizontal plane where the main drone 10 is located. Here, with intersection point M as the origin (center), consider the coordinate systems of each position. The perpendicular AC is in the viewpoint direction of camera 21 (vertical field of view θ). fvfov (Indicates the direction of the center)

[0063] Pixel coordinate x pixel ,y pixel From third-person view coordinate x m (t), y m The conversion to (t) is expressed by the following formula.

number

number

[0064] The variables h and a in equations 2 and 3 are expressed by the following equations.

number

number

[0065] Here, h=AM and a=RM. f (t) is relative altitude, θ fvfov This is the vertical field of view of camera 21. Of these, y f (t) is obtained from the altitude information of the sensor 12 of the main drone 10 and the sensor 22 of the sub-drone 20, and θ fvfov This is known information about camera 21, and h and a can be easily calculated from this information.

[0066] <Distance x in actual coordinates> f (t), z f Derivation of (t) In order to determine the actual relative positions of the main drone 10 and the sub-drone 20, it is necessary to convert the third-person view coordinates to real-world coordinates. The position calculation unit 31b uses the following formula to calculate the third-person view coordinate x m ,y m The distance obtained from this is expressed in real coordinates (real space, real world) x f (t), z f Convert to the distance obtained from (t).

[0067]

number

number

[0068] In equations 6 and 7, φ can be expressed by the following equation.

number

[0069] In other words, real coordinates (real space, real world) x f (t), z f The distance obtained from (t) is the horizontal distance between the main drone 10 and the sub-drone 20, and is in real coordinate x f (t), z f The distance obtained from (t) is calculated using the pixel coordinates, the angle φ of the camera 21 of the sub-drone 20 with respect to the perpendicular AC, and the vertical field of view θ of the camera 21. fvfov and the altitude difference y between the main drone 10 and the sub-drone 20 f It is calculated using (t).

[0070] In deriving equations 2 to 8, the pixel coordinates of the main drone 10 are obtained by image recognition processing, and the relative altitude y between the drones is f (t) is obtained from sensors 12 and 22. Also, the tilt of camera 21 (target camera angle θ) f (t) and the vertical field of view θ of camera 21 fvfov This is known. Thus, the pixel coordinate x pixel ,y pixel The distance and real coordinates (real space, real world) x are obtained from this. f (t), z f The distance (relative distance) obtained from (t) can be easily calculated with a small number of variables without complex calculations.

[0071] Once the coordinates of the main drone 10 and the sub-drone 20 are determined, the optimization unit 31a adjusts the positions of the main drone 10 and the sub-drone 20 so that they have the optimal relative position. Furthermore, after achieving the optimal relative position, the optimization unit adjusts the angle θ of the camera 21. f (t) and target relative angle φ f Adjust (t) to the optimal angle.

[0072] [3-3. Marker Generation Section] Distance x calculated by position calculation unit 31b f (t), z f Based on (t), information that enables understanding and prediction of the space surrounding the aircraft is visualized. The marker generation unit 31c generates markers for display on the overhead view image.

[0073] Figure 5 shows the top-perspective view (TPV) image from Figure 2 with markers added. In Figure 5, a triangular marker 42 indicating the orientation of the main drone 10, a vertical line marker 44 indicating the height, a weighted marker 43 indicating the shooting area of ​​the main drone 10, and circular markers 41 (e.g., radius 5m and radius 8m) indicating the distance from the main drone 10 are superimposed on the TPV image. Such a highly visible image allows the pilot to quickly understand and predict the space around the aircraft.

[0074] In this embodiment, the operator overlays markers onto an overhead view image using augmented reality (AR) to display spatial information around the drone. For example, the cast shadow method may be used for the AR overlay.

[0075] To understand the aircraft's status at the time of image acquisition, information regarding the orientation and altitude of the main drone 10 is necessary.

[0076] The orientation (direction of travel) of the main drone 10 is indicated using a triangular marker (second marker) 42, as an example. The method using triangles has a lower cognitive load compared to the method using arrows. An isosceles triangle is preferable, and the direction pointed to by the vertex angle of the isosceles triangle is the direction of travel of the main drone 10. Furthermore, if the main drone 10 is not hovering but is moving, the size of the triangular marker 42 may be changed according to the speed of the main drone 10. Note that any marker that allows for an intuitive understanding of the direction of movement is acceptable, not just the triangular marker 42.

[0077] The altitude of the main drone 10 is indicated by a vertical line marker (fourth marker) 44. The vertical line is a straight line connecting the reference plane and a predetermined part of the main drone 10 (for example, the center of gravity of the main drone 10), and the reference plane and the vertical line are perpendicular. In other words, the upper end of the line represents the position of the main drone 10. The reference plane is, for example, the ground surface, and may also be the distance (height) to the nearest vertically below the main drone 10, not limited to altitude. The distance to such structures is obtained by a sensor 12 or the like. Furthermore, the vertical line marker 44 may display a predetermined scale, or its color may change, such as turning red when the distance falls below a certain threshold. Numerical information regarding altitude may also be displayed within the TPV. Note that the vertical line marker 44 is not limited to a vertical line; it may be any form that intuitively conveys information about altitude.

[0078] In this example, the shooting range of the main drone 10 is indicated using a marker (third marker) 43 on a pyramidal pyramid. The base of the pyramidal pyramid represents the shooting range of the main drone 10, and the vertices represent the position of the main drone 10. In other words, the base of the pyramidal pyramid is the rectangular frame of the area image (FPV) captured by the main drone 10, as will be described later using Figure 5. Furthermore, the marker 43 on the pyramidal pyramid represents the area being photographed by the camera 11, and the lines connecting the camera 11 to the four corners of the marker 43 on the pyramidal pyramid may represent the space being photographed by the camera 11. Note that the lines connecting camera 11 to the four corners of the marker 43 on the truncated pyramidal pyramid do not need to be displayed.

[0079] Of the information available about the main drone 10, the most important is the distance from the main drone 10. Because it is difficult for the pilot to grasp the actual positional relationship and angle between the main drone 10 and the sub-drone 20 from the overhead view image itself, it is easy to misjudge how far the main drone 10 is from objects that appear in the overhead view image. According to this embodiment, the pilot can grasp the position of the main drone in real coordinates, so they can accurately grasp how far away objects and places that appear in the image are from the main drone 10.

[0080] The distance from the main drone 10 is indicated by a circular marker (first marker) 41 centered on the main drone 10. The circular marker 41 can be displayed at any distance selected by the operator from the main drone 10. Furthermore, multiple distances, such as 3m and 5m, can be displayed using multiple circular markers 41. Furthermore, the marker 41 indicating the distance from the main drone 10 is not limited to a circular marker; any type of marker that allows for intuitive understanding of the distance from the main drone 10 may be used.

[0081] In conventional methods, it is difficult to accurately determine the position (distance) from the main drone 10 to surrounding objects. For example, in pesticide spraying, it is not possible to spray to the target location, and in product delivery, it is not possible to deliver products to the recipient at the optimal location. However, in the present invention, the positional relationship between the main drone 10 and the sub-drone 20 falls within a predetermined range, and the positions of the main drone 10 and the sub-drone 20 are set in a coordinate system, so it is possible to display a marker 41 that accurately indicates the distance from the main drone 10.

[0082] Furthermore, in addition to understanding the current distance in the surrounding space of the main drone 10, it is desirable to be able to predict the position of the main drone 10 after a predetermined time. To predict the position of the main drone 10, information is needed not only about the distance between the main drone 10 and objects in the surrounding environment, but also about the expected destination of the main drone 10.

[0083] The controller 30 receives images from the main drone 10 and the sub-drone 20 in real time. However, since humans require time to make a decision called selective reaction time, the planned destination of the drone 10 must be within a range that takes into account the operator's selective reaction time.

[0084] The intended destination of the main drone 10 is indicated by a circular marker (first marker) centered on the main drone 10, similar to the distance marker 41, for reasons of cognitive load and because the drone can move in all directions. In this embodiment, the radius of the circle is defined as the direction of travel L shown by the following equation.

[0085]

number

[0086] v(t) is the velocity of the main drone 10 at time t, and 1.5[s] is the value proposed in the literature (Hick, WE [Quarterly Journal of Experimental Psychology 1952]) as the selection reaction time.

[0087] The expected destination of the main drone 10 may be displayed by multiple circles by setting multiple time t values. Furthermore, if the main drone 10 moves not only horizontally but also vertically, a corresponding spherical marker may be used, or the predicted position at time t may be displayed within the TPV using the drone's marker, or the movement trajectory may be shown.

[0088] The various markers 41-44 within the TPV can be intentionally hidden by using a function that allows users to select whether or not to display them, thus preventing the display of anything other than what is necessary. Having multiple markers can sometimes make it difficult for the pilot to see them. In such cases, it might be better to allow the pilot to select only the markers they want to display, thus enabling them to operate the system without stress. Alternatively, the circular marker 41 indicating distance and the vertical line marker 44 indicating height may be automatically displayed when they fall below a predetermined threshold, for the safety of the main drone 10 and its surroundings.

[0089] To display spatial information in AR, it is necessary to convert real-space coordinates to pixel coordinates. The position calculation unit 31b calculates the pixel coordinate values ​​using the inverse functions of equations 6 to 8. The information to be drawn using the cast shadow method is the y of equations 4 and 5. f You can substitute the altitude of the secondary drone 20 into (t). Alternatively, to draw concentric circles, you can convert the position coordinates of four points equidistant from the main drone 10 into pixel coordinates, and then apply the perspective projection transformation matrix using those four points to a perfect circle.

[0090] [3-4.Display section] The display unit 35 of the controller 3 displays the area image and the overhead view image on the screen of the display device 35. The display unit 35 further displays the triangular marker 42, vertical line marker 44, weighted marker 43, and circular marker 41 generated by the marker generation unit 31c within the overhead view image.

[0091] Figure 6 shows an example of the display device 35 showing both the FPV and the TPV with markers. In this embodiment, the two types of images are arranged vertically, with the upper image being an area image and the lower image being an overhead view image with markers. Arranging the images vertically is less likely to cause confusion for the pilot than arranging them horizontally and parallel to each other. The display device 35 may display two images vertically on one screen, or it may have two screens connected by a hinge mechanism, allowing the upper and lower screens to be tilted towards the pilot.

[0092] [4. Drone control operation] Figure 7 is a flowchart showing an example of control performed by the controller 30. This flow starts when the main drone 10 and the sub-drone take off (for example, when the ON signal from the activation switch is received) and is performed by the controller 30 until both drones 10 and 20 finish their flight (for example, until the OFF signal is received from the activation switch).

[0093] As illustrated in Figure 7, the CPU 31 of the controller 30 acquires information on the main drone 10 and the sub-drone 20 via the IF unit 34 (step S1). The information on the main drone 10 includes an area image (area information) captured by the camera 11 of the main drone 10, and the orientation, altitude, and speed of the drone itself measured by the sensor 12. The information on the sub-drone 20 includes an image (overhead view image) of the main drone 10 captured by the camera 21 from an overhead view, the tilt of the camera 21, and the orientation and altitude of the drone itself measured by the sensor 12.

[0094] After the main drone 10 begins flight, the secondary drone 20 follows the main drone 10 and begins flight. Once it reaches the target position, it automatically moves to a predetermined position where the main drone 10 can be seen within an overhead view using the method described above.

[0095] The position calculation unit 31b of the controller 30 detects the main drone 10 in the overhead view image and obtains the pixel coordinates of the main drone 10 (step S2). The pixel coordinates are in a coordinate system centered on point M shown in Figure 3.

[0096] Next, the position calculation unit 31a calculates the horizontal distance between the main drone 10 and the sub-drone 20 (step S3). The horizontal distance is the distance between the drones in actual coordinates x f (t), z f (t) is the distance x in real coordinates. f (t), z f (t) is the angle φ of the camera 21 of the auxiliary drone 20 with respect to the perpendicular AC, and the vertical field of view θ of the camera 21. fvfov and the altitude difference y between the main drone 10 and the sub-drone 20 f It is calculated using (t).

[0097] After step S3, two processes that can be executed simultaneously are performed. First, the marker generation process, which is the flow on the left, is described. The marker generation unit 31c generates markers that indicate information about the main drone 10 to be displayed in the overhead view coordinates (step S4). The markers include a triangular marker 42 indicating the orientation of the main drone 10, a circular marker 41 indicating the distance from the main drone 10, a weighted marker 43 indicating the shooting area of ​​the main drone 10, and a vertical line marker 44 indicating the height of the main drone 10.

[0098] The display unit 35, specifically the marker generation unit 31c, overlays the generated markers onto the overhead view image of the sub-drone 20 and displays (outputs) them together with the area image of the main drone 10 on the screen of the display device 35 (step S5).

[0099] Next, we describe the position control process of the sub-drone 20, which is the other process after step S3. The optimization unit 31a calculates the optimal position of the sub-drone 20 (step S6). The optimal position is calculated using the objective function represented by Equation 1, under constraint conditions i to vii that realize the aforementioned optimal position conditions (a) to (c). The optimization unit 31a calculates the current three-dimensional positions of the main drone 10 and the sub-drone 20, the angle of the camera 21 of the sub-drone 20, and the target relative angle φ. f By using the primary dual interior point method with (t), etc., the optimal placement position of the secondary drone 20 (three-dimensional target position), the angle of the camera 21, and the target relative angle are calculated.

[0100] The optimization unit 31a adjusts the position of the sub-drone 20 based on the calculated control values ​​(step S7). The optimization unit 31a transmits a control signal to the sub-drone 20's communicator 23 via the IF unit 34, and the sub-drone 20's controller 24 moves the aircraft to the three-dimensional target position. At that position, the camera 21's angle and the target and relative angles are adjusted to the optimal angles.

[0101] [5. Experimental Examples] [5-1. Experiment Details] An experiment was conducted to verify the extent to which the drone system 100 of the present invention improved the operator's spatial awareness (SA).

[0102] In the experiment, to minimize the influence of learning and training effects on drone operation, the pilots were six inexperienced individuals with between 1 hour and 10 hours of drone operation experience. The experiment was conducted outside of densely populated areas. To avoid the effects of direct sunlight on the pilot, a tent was set up, and the pilot operated the drone from inside the tent. The flight was conducted within visual line of sight (visual line of sight) (because by law, unskilled pilots are prohibited from flying beyond visual line of sight).

[0103] In the experiment, the primary drone used was a DJI Mavic 2 Pro, and the secondary drone was a Parrot Anafi 4K. The experiment involved performing two types of tasks that required movement in a direction different from the orientation of the drone's camera (movement in the blind spot of the FPV) and detailed positional awareness. The tasks were: 1. Nose-in-Circle task (a task of circling a subject while displaying the subject on the camera's FPV), which is easy to determine if the drone understands the state of its surroundings (SA Level 2); and 2. High-speed movement task (a task of moving between multiple poles as quickly as possible), which is easy to determine if the drone can predict the state of its surroundings (SA Level 3).

[0104] In the Nose-in-Circle task, the main drone continuously circles the subject while maintaining a distance of 5 meters from it. For comparison, we used three methods with different interfaces: FPV only, FPV plus dynamic third-person view without AR (AutoTPV), and the drone system 100 method of the present invention (AR-BirdView). The evaluation was based on the average error distance between the subject and the main drone.

[0105] In the high-speed movement task, the main drone moves between three types of poles A, B, and C. Pole B is positioned 20m away from pole A, and pole C is positioned 20m away from pole B, perpendicular to the line of poles A and B. In this task, the main drone was instructed to move as quickly as possible and stop 5-8 meters from the pole. To reduce the learning effect, three possible routes were prepared to visit poles A-C: A → B → C → B → A, B → C → B → A → B, and C → B → A → B → C. The task was performed using a different route for each interface. For comparison, we used three methods: FPV only, a third-person view from directly above with AR superimposed (displaying the radius distance centered on the main drone) (AR-TopView), and the method of the present invention (AR-BirdView). In the experiment, quantitative evaluations were made using metrics such as the total average time required to travel along the above route, the average percentage of times the drone was able to stop within the designated area (5-8 m from the pole) (task success rate), and the distance traveled by the main drone until the task was completed.

[0106] The pilots practiced each of the three control interfaces for about 10 minutes. In addition to linear motion of the drone, they also practiced turning and maneuvering. Once they became familiar with each interface, they moved on to the respective experimental tasks.

[0107] Each pilot practiced the Nose-in-Circle task for 5-10 minutes, followed by an experiment on the Nose-in-Circle task. Subsequently, they practiced and verified routes for the high-speed movement task for 5-10 minutes each, followed by an experiment on the high-speed movement task. In addition, for each stage, the pilots were given questionnaires regarding the extent to which they were able to perceive the surrounding environment of the main drone (spatial awareness), the extent to which they understood the positional relationship between the main drone and the subject or pole (spatial understanding), the ease of creating a flight plan for the main drone (motion planning), their level of anxiety regarding the direction of movement (anxiety), and the degree to which they needed to concentrate on piloting (concentration), thereby obtaining subjective data from the pilots.

[0108] [5-2.Results] [5-2-1. Nose-in-Circle Task] In the Nose-in-Circle task, when the interface was FPV only, the pilot determined the distance based on the size of the subject displayed on the main drone's camera screen. In AutoTPV, the distance was determined based on the distance between the main drone and the secondary drone, in addition to the image from the main drone's camera. In the AR-BirdView of this invention, the distance was determined based on the distance between the grid lines displayed in the overhead view image from the secondary drone and the poles, in addition to the image from the main drone's camera.

[0109] The average error in the distance between the subject and the main drone was calculated for each interface. As a result, when the interface was FPV only, an average error of around 1.28m occurred. When the interface was AutoTPV, an average error of around 0.85m occurred. With the interface of the present invention, the average error range fell within approximately 0.37m.

[0110] Furthermore, when we surveyed pilots about the level of concentration and other workload they experienced during operation with each interface, FPV showed the highest workload. In particular, spatial awareness, spatial understanding, motion planning, and anxiety were scored more than twice as bad as AutoTPV and the method of the present invention. On the other hand, when comparing the workload of AutoTPV and the interface of the present invention, the scores were similar, but the interface of the present invention showed significantly better performance in terms of spatial understanding and motion planning.

[0111] [5-2-2. High-speed movement task] In high-speed movement tasks, if the interface is FPV only, the pilot must determine the direction of movement based solely on the video of the direction of travel captured by the main drone itself. In addition, AR-TopView provides not only the video from the main drone's camera but also a video of the main drone viewed from directly above. In the AR-BirdView of this invention, the secondary drone receives not only the video from the main drone's camera but also a video that the secondary drone has focused on displaying, with grid lines indicating the distance to the direction the main drone is moving.

[0112] In high-speed movement tasks, the average total time required to travel along a path was approximately 33-34 seconds with FPV only, and approximately 35-36 seconds with AR-TopView. On the other hand, the method of the present invention achieved the fastest travel time of 31-32 seconds. The task success rate was approximately 75% with FPV only, approximately 71% with AR-TopView, and approximately 88% with the method of the present invention, demonstrating a significant improvement in the success rate when using the method of the present invention. Furthermore, the distance traveled to complete the task was approximately 103m with FPV only, approximately 101m with AR-TopView, and approximately 91m with the method of the present invention, demonstrating that the method of the present invention enables efficient flight.

[0113] In a survey regarding the level of concentration and other workload placed on pilots during high-speed movement tasks, similar to the Nose-in-Circle task, FPV showed the highest workload. In particular, spatial awareness, spatial understanding, motion planning, and anxiety scores were 2 to 4 times worse than those of the present invention. In all aspects, the present invention's method was evaluated as being 1.2 times superior to AR-TopView. Significant differences were observed, especially in the items of spatial understanding, motion planning, and anxiety. The present invention's method facilitates spatial understanding, making it easier to anticipate flight motion plans and understand the surrounding environment, thus enabling pilots to operate without anxiety even when moving at high speeds.

[0114] [5-3. Conclusion] For both the Nose-in-Circle task and the high-speed movement task, the method of the present invention proved to be the most performance-oriented and least resource-intensive. The method of the present invention exhibits low error even in the nose-in-circle task, thus satisfying SA Level 2, which indicates an understanding of the state around the aircraft. Furthermore, since the task success rate is nearly 100% even in the high-speed movement task, it can be said that the system has reached SA Level 3, which indicates an ability to predict the state around the aircraft.

[0115] [6. Effects] (1) The drone system 100 of this embodiment includes a main drone 10 equipped with a camera 11, a sub-drone 20 equipped with a camera 21 that captures images of the main drone 10 from an overhead perspective, and a controller 30 that controls the main drone 10 and the sub-drone 20. The controller 30 includes a display unit 35 that displays the overhead perspective image captured by the camera 21, and has a function to display a circular marker 41 indicating the distance from the main drone 10 within the overhead perspective image. By displaying an overhead view image captured from a bird's-eye perspective of the main drone 10, the operator can check blind spots around the main drone 10. Furthermore, distance information is visualized within the overhead view image using circular markers 41, which was previously difficult to do, making it easy to predict the future position and status of the main drone 10. Therefore, the main drone 10 can perform tasks with high precision. Furthermore, even in situations such as beyond visual line of sight (BVLOS) flight, the status of the main drone 10 can be understood from the overhead view image, allowing for safe and accurate work.

[0116] (2) The controller 30 may have a function to further display a triangular marker 42 indicating the orientation of the main drone 10 within the overhead view image. The triangular marker 42 visualizes the orientation of the main drone 10, making it easy to understand the drone's target direction. In particular, in tasks where aerial photography is performed while tracking a moving object, the orientation of the camera 11 and the direction of the drone's movement are often different, making it difficult for the operator to notice obstacles in the direction of the main drone 10's movement. However, the display of the triangular marker 42 alerts the operator to the direction of the main drone 10's movement, thus reducing the risk of the main drone 10 coming into contact with or colliding with obstacles.

[0117] (3) The controller 30 may have a function to further display a weighted marker 43 in the overhead view image that indicates the area being captured by the camera 11. By making the imaging area of ​​the camera 11 visible with the weighted marker 43, the operator can more easily grasp the object being photographed by the main drone 10, and thus be able to accurately perform tasks in tasks where a first-person view is important, such as aerial photography and inspection. In addition, the display of the weighted marker makes it easier to match the overhead view image with the area image.

[0118] (4) The position calculation unit 31b of the controller 30 converts the pixel coordinates of the main drone 10 as seen in the overhead view image into real-space coordinates, thereby avoiding update delays like those that occur when using GPS coordinates, and providing highly accurate coordinates and distances without large errors.

[0119] (5) The controller 30 moves the sub-drone 20 to the three-dimensional target position where the sub-drone is to be placed, under the conditions that the main drone 10 is contained within the overhead view image, the area captured by the camera 11 is contained within the overhead view image, and the range in which the main drone 10 moves over a predetermined time is contained within the overhead view image, and the angle θ of the camera 21 f (t) and target relative angle φ f The system further includes an optimization unit 31a that controls (t). The optimization unit 31a moves the sub-drone 20 to the three-dimensional target position so as to satisfy the above conditions (a) to (c), thereby accurately grasping the status of the main drone 10 and enabling the constant display of various information within the overhead view image.

[0120] (6) The optimization unit 31a determines the three-dimensional positions of the main drone 10 and the sub-drone 20, and the angle θ of the camera 21 of the sub-drone. f (t) and target relative angle φ f The three-dimensional target position where the sub-drone 20 is positioned, the angle of the second imaging means, and the relative angle to the target are determined using the primary dual interior point method based on (t), and controlled using cascaded PID. By optimizing using the primary dual interior-point method and calculating the control variables using cascaded PID, the secondary drone 20 can be controlled more accurately.

[0121] (7) The controller 30 determines the coordinates x and y of the main drone 10 relative to the sub-drone 20 in the plane, the angle φ of the sub-drone 20's camera 21 with respect to the perpendicular AC, and the vertical field of view θ of the camera 21. fvfov The calculation is performed using the altitude difference Alt between the main drone 10 and the secondary drone 20. Perpendicular line AC of camera 21, vertical field of view θ of camera 21 fvfov Furthermore, since the altitude difference Alt between the main drone 10 and the sub-drone 20 can be easily obtained from the values ​​of the camera 21 and sensor 22, respectively, the coordinates x and y of the main drone 10 in the plane relative to the sub-drone 20 can be easily calculated simply by calculating the angle φ of the sub-drone 20's camera 21 with respect to the perpendicular AC.

[0122] (8) The main drone 10, the sub-drone 20, and the controller 30 are equipped with communication means 13, 23, and 34. The main drone 10's communicator 13 transmits information about the area captured by the camera 11 (area image), the orientation, altitude, and speed of the main drone to the controller 30's IF unit 34. The sub-drone 20's communicator 23 transmits an overhead view image, the tilt of the sub-drone 20's camera 21 (angle in the focal field of view), and the orientation and altitude of the sub-drone 20 to the controller 30's IF unit 34. The controller 30's IF unit 34 transmits signals to the communicators 13 and 23 of the main drone 10 and sub-drone 20 to control the main drone 10 so that it fits within the overhead view captured by the sub-drone 20's camera 21. The main drone 10 and the sub-drone 20 each transmit the above-mentioned information to the controller 30, so that the position calculation unit 31b of the controller 30 can calculate the horizontal distance x,y between the main drone 10 and the sub-drone 20, the marker generation unit 31c can display markers on the overhead view image, and the optimization unit 31a can calculate the optimal position of the sub-drone 20. Furthermore, the controller 30 transmits the above control signals to the main drone 10 and the sub-drone 20, causing the sub-drone 20 to move to an optimal position, thereby enabling the capture of an appropriate overhead view image corresponding to the speed of the main drone 10.

[0123] [7. Variant]

[0124] In addition to the above control, further control may be necessary in situations such as when the main drone 10 goes under something, or when an obstacle causes it to enter the blind spot of the secondary drone 20. Since the method of the present invention determines the placement position using an optimization function, such problems can be addressed by introducing terms corresponding to the main drone 10 going under something or the secondary drone 20 avoiding obstacles into the constraints and objective function of the optimization function.

[0125] Figure 8 shows the positional relationship between the drone and obstacles. (a) is a view of the drone from the side, and (b) is a third-person view from a secondary drone. Add the following equation 10 to the constraints in the optimization method section mentioned above.

number

[0126] <00006​​​​​​​​​​​​​​​​​​​​​​​​​​​​According to the modified method, by using equations 11 and 12, the controller 30 can control the sub-drone 20 so that no obstacles enter between the main drone 10 and the sub-drone 20.

[0129] [8. Other] The types and shapes of markers described above are merely examples and are not limited to those mentioned above. For example, additional markers may be added to indicate specific objects necessary for spatial recognition.

[0130] Furthermore, the secondary drone 20 may be used as a relay station. This makes it possible to extend the range of the main drone 10's radio-controlled flight.

[0131] In this embodiment, two drones were used, but multiple drones may be used to generate and display a three-dimensional image by combining images captured by the drone group. This can further improve the operator's perception of the surrounding space of the main drone 10.

[0132] The main drone 10 and the sub-drone 20 can be swapped alternately. This allows for desired imaging according to the environment. In this case, the cameras, sensors, and communication devices of each drone 10 and 20 in the above embodiment can be made to common specifications, and the controller can be made to have control functions for both the main and sub-drones. This allows for the above effect to be obtained while standardizing parts. When communicating with the controller 30, the communication code is changed when the drones are swapped to notify the controller 30 of the drone swap. In this way, the configuration and functions of the controller do not need to be changed.

[0133] In this embodiment, the display device 35 is given as an example of a monitor shape, but it is not limited to this, and may be, for example, AR glasses or the like. [Explanation of Symbols]

[0134] 10: Main Drone 11: Camera (first imaging means) 12: Sensor 13: Communication device (first means of communication) 14: Controller 20: Sub-drone 21: Camera (second imaging means) 22: Sensor 23: Communication device (second means of communication) 24: Controller 30: Controller 31: CPU (Computer) 31a: Optimization unit (control unit) 31b: Position calculation section 31c: Marker generation unit 32: Memory 33: Storage device 34: IF section (controller's communication method) 35:Display device (display section) 41: Circle marker (first marker) 42: Triangular marker (second marker) 43: Pyramidal pyramidal marker (third marker) 44: Vertical line marker (fourth marker) 100: Drone System

Claims

1. A drone system comprising a main drone equipped with a first imaging means, a secondary drone equipped with a second imaging means for imaging the main drone from an overhead perspective, and a controller for controlling the main drone and the secondary drone, The controller includes a display unit that displays an area image captured by the first imaging means and an overhead view image captured by the second imaging means. The controller has a function to superimpose a first marker onto the overhead view image to determine the distance from the main drone. Drone system.

2. The controller has a function to further superimpose a second marker indicating the orientation of the main drone onto the overhead view image. The drone system according to claim 1.

3. The controller has a function to further superimpose a third marker indicating the area captured by the first imaging means onto the overhead view image. The drone system according to claim 1 or 2.

4. The controller has means for converting the pixel coordinates of the main drone as seen in the overhead view image into coordinates in real space. The drone system according to claim 1 or 2.

5. The aforementioned controller, Under the conditions that the main drone is contained within the overhead view image, the area captured by the first imaging means is contained within the overhead view image, and the range over which the main drone travels in a predetermined time is contained within the overhead view image, The system further includes a control unit that moves the sub-drone to the three-dimensional target position where the sub-drone is positioned, and controls the angle of the second imaging means and the relative angle to the target. The drone system according to claim 1 or 2.

6. The control unit determines the three-dimensional target position where the sub-drone is positioned, the angle of the second imaging means, and the target relative angle using a principal-dual interior point method based on the three-dimensional positions of the principal drone and the sub-drone, the angle of the second imaging means of the sub-drone, and the target relative angle, and controls them using a cascaded PID. The drone system according to claim 5.

7. The controller calculates the relative distance between the main drone and the sub-drone using the angle of the sub-drone with respect to the perpendicular of the second imaging means, the field of view of the second imaging means, and the altitude difference between the main drone and the sub-drone. The drone system according to claim 1 or 2.

8. The main drone, the sub-drone, and the controller are equipped with communication means. The communication means of the main drone transmits information about the area captured by the first imaging means, the orientation, altitude, and speed of the main drone to the communication means of the controller. The communication means of the sub-drone transmits the overhead view image, the tilt of the second imaging means of the sub-drone, the orientation and altitude of the sub-drone to the communication means of the controller. The controller's communication means transmits a signal to the communication means of the main drone and the sub-drone to control the main drone so that it is positioned within the overhead view captured by the sub-drone's second imaging means. The drone system according to claim 1 or 2.

9. The controller controls the system to prevent obstacles from being placed between the main drone and the sub-drone. The drone system according to claim 1 or 2.

10. A computer is provided in a controller that controls a main drone equipped with a first imaging means and a sub-drone equipped with a second imaging means that images the main drone from an overhead perspective. The area image captured by the first imaging means and the overhead view image captured by the second imaging means are displayed on the display device. A first marker for determining the distance from the main drone is superimposed on the overhead view image of the display device. A drone control program that executes a process.

11. A computer provided in a controller that controls a main drone equipped with a first imaging means and a sub-drone equipped with a second imaging means that images the main drone from an overhead perspective, The area image captured by the first imaging means and the overhead view image captured by the second imaging means are displayed on the display device. A first marker for determining the distance from the main drone is superimposed on the overhead view image of the display device. A drone control method for executing a process.

Images