Unmanned aerial vehicle (UAV) monitoring system, UAV monitoring method, and UAV monitoring program

Abstract

This system reduces the impact on the overall flight path of multiple unmanned aerial vehicles (UAVs) if an abnormality occurs in the flight status of an UAV positioned in a critical position in a formation flight. [Solution] The unmanned aircraft monitoring system 10 acquires telemetry information indicating the flight status of unmanned aircraft during formation flight, associated with individual identification information that identifies each unmanned aircraft, determines whether there is an abnormality in the flight status of the unmanned aircraft based on the information, the individual identification information includes position information regarding the position of the unmanned aircraft in the formation, and determines whether the unmanned aircraft is positioned in an important position in the formation based on the individual identification information, and transmits a command to the radio aircraft in formation flight to replace the unmanned aircraft that has been determined to be positioned in an important position and has been determined to have an abnormality with an unmanned aircraft that has been determined not to be positioned in an important position and has been determined to have no abnormality.

Description

Technical Field

[0001] The present invention relates to an unmanned aircraft monitoring system, an unmanned aircraft monitoring method, and an unmanned aircraft monitoring program, and more particularly to an unmanned aircraft monitoring system, an unmanned aircraft monitoring method, and an unmanned aircraft monitoring program suitable for monitoring unmanned aircraft during formation flight.

Background Art

[0002] In recent years, the technology of unmanned aircraft such as drones has advanced, and it has become possible to use multiple unmanned aircraft in formation flight (for example, see Patent Document 1). An example of flying multiple unmanned aircraft in formation flight is a drone show. A drone show refers to a show in which multiple drones in formation flight fly synchronously and combine light, sound, and further video.

[0003] In a drone show, if an abnormality occurs in the flight state of some unmanned aircraft and they drop out, there is a risk that the description drawn by the multiple unmanned aircraft in cooperation will be disrupted or become a description different from the plan. In particular, when an abnormality occurs in an unmanned aircraft placed in an important position within the formation and it drops out, the impact on the description is significant.

[0004] In the flight control method for flying multiple aircraft in formation disclosed in Patent Document 1, no consideration was given to the occurrence of abnormalities in the unmanned aircraft during formation flight.

Prior Art Documents

Patent Documents

[0005] [[ID=~30]]<~

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] Therefore, the present invention aims to provide an unmanned aerial vehicle monitoring system 10, an unmanned aerial vehicle monitoring method, and an unmanned aerial vehicle monitoring program that can mitigate the impact on the depiction created by multiple unmanned aerial vehicles when an abnormality occurs in the flight state of an unmanned aerial vehicle positioned in an important position in formation flight. [Means for solving the problem]

[0007] In other words, the first embodiment of the unmanned aircraft monitoring system is an unmanned aircraft monitoring system that monitors the flight status of multiple unmanned aircraft in formation flight, and is characterized by comprising: a telemetry information acquisition unit that acquires telemetry information indicating the flight status of the unmanned aircraft in formation flight transmitted from each of the multiple unmanned aircraft, associated with individual identification information that is assigned to the multiple unmanned aircraft in advance and identifies each individual unmanned aircraft; an abnormality determination unit that determines whether or not there is an abnormality in the flight status of the unmanned aircraft based on the telemetry information; an important position determination unit that determines whether or not the unmanned aircraft is positioned in an important position in the formation, based on the individual identification information, and the individual identification information includes position information relating to the position of the unmanned aircraft in the formation; and a transmission unit that transmits a command to the unmanned aircraft in formation flight to replace an unmanned aircraft that has been determined to be positioned in an important position and has been determined to have an abnormality with an unmanned aircraft that has been determined not to be positioned in an important position and has been determined to have no abnormality.

[0008] The second embodiment further comprises an image acquisition unit that acquires images captured by a camera that images the patterns drawn by multiple unmanned aircraft in formation flight, a pattern anomaly determination unit that determines whether or not there is a pattern anomaly based on the captured images, and an anomaly extraction unit that extracts individual identification information of the unmanned aircraft causing the anomaly when it is determined that the pattern is anomaly.

[0009] A third embodiment is an unmanned aerial vehicle monitoring system according to the first embodiment, wherein the telemetry information may include at least one of the following: altitude, temperature, battery level, attitude, and position coordinates of the unmanned aerial vehicle.

[0010] A fourth embodiment may further include a learning model acquisition unit in the unmanned aerial vehicle monitoring system according to the second embodiment, which acquires a learning model that learns normal depictions in advance, determines whether or not there are any abnormalities in the depictions included in the captured image, and outputs individual identification information of the unmanned aerial vehicle that is causing the abnormality when it is determined that there is an abnormality in the depiction, and the depiction abnormality determination unit may use the learning model to determine whether or not there are any abnormalities in the depictions included in the captured image, and the abnormality extraction unit may use the learning model to extract individual identification information of the unmanned aerial vehicle that is causing the abnormality included in the captured image.

[0011] A fifth embodiment is an unmanned aerial vehicle monitoring system according to the fourth embodiment, wherein the learning model pre-learns the shape and brightness of the contours of normal depictions, determines whether or not there are any abnormalities in the depictions included in the captured image, and outputs individual identification information of the unmanned aerial vehicle that caused the abnormality if it is determined that there are abnormalities in the depictions included in the captured image.

[0012] The unmanned aircraft monitoring method according to the sixth embodiment is an unmanned aircraft monitoring method used in an unmanned aircraft monitoring system that monitors the flight status of multiple unmanned aircraft in formation flight, characterized in that the unmanned aircraft monitoring system is made to perform the following steps: a telemetry information acquisition step of acquiring telemetry information indicating the flight status of the unmanned aircraft in formation flight, which is transmitted from each of the unmanned aircraft and is associated with individual identification information that is pre-assigned to the multiple unmanned aircraft and identifies each individual unmanned aircraft; an abnormality presence / absence determination step of determining whether or not there is an abnormality in the flight status of the unmanned aircraft based on the telemetry information; an important position determination step of determining whether or not the unmanned aircraft is positioned in an important position in the formation, based on the individual identification information, and the individual identification information includes position information regarding the position of the unmanned aircraft in the formation; and a transmission step of transmitting a command to the radio aircraft in formation flight to replace an unmanned aircraft that has been determined to be positioned in an important position and has been determined to have an abnormality with an unmanned aircraft that has been determined not to be positioned in an important position and has been determined to have no abnormality.

[0013] The seventh embodiment of the unmanned aircraft monitoring program is an unmanned aircraft monitoring program used in an unmanned aircraft monitoring system that monitors the flight status of multiple unmanned aircraft in formation flight, and is characterized in that the unmanned aircraft monitoring system implements: a telemetry information acquisition function that acquires telemetry information indicating the flight status of unmanned aircraft in formation flight transmitted from each of the unmanned aircraft, associated with individual identification information that is pre-assigned to the multiple unmanned aircraft and identifies each individual unmanned aircraft; an abnormality determination function that determines whether or not there is an abnormality in the flight status of the unmanned aircraft based on the telemetry information; an important position determination function that determines whether or not an unmanned aircraft is positioned in an important position in the formation, based on the individual identification information, and the individual identification information includes position information regarding the position of the unmanned aircraft in the formation; and a transmission function that transmits a command to the unmanned aircraft in formation flight to replace an unmanned aircraft that has been determined to be positioned in an important position and has been determined to have an abnormality with an unmanned aircraft that has been determined not to be positioned in an important position and has been determined to have no abnormality. [Effects of the Invention]

[0014] The unmanned aerial vehicle monitoring system according to the present invention is an unmanned aerial vehicle monitoring system that monitors the flight status of multiple unmanned aerial vehicles during formation flight, and is characterized by comprising: a telemetry information acquisition unit that acquires telemetry information indicating the flight status of the unmanned aerial vehicles during formation flight, which is transmitted from each of the unmanned aerial vehicles and is associated with individual identification information that is assigned to the multiple unmanned aerial vehicles in advance and identifies each individual unmanned aerial vehicle; an abnormality determination unit that determines whether or not there is an abnormality in the flight status of the unmanned aerial vehicles based on the telemetry information; an important position determination unit that determines whether or not the unmanned aerial vehicle is positioned in an important position in the formation, based on the individual identification information, which includes position information regarding the position of the unmanned aerial vehicles in the formation; and a transmission unit that transmits a command to the unmanned aerial vehicles in formation flight to replace an unmanned aerial vehicle that has been determined to be positioned in an important position and has been determined to have an abnormality with an unmanned aerial vehicle that has been determined not to be positioned in an important position and has been determined to have no abnormality. Therefore, if an abnormality occurs in the flight status of an unmanned aerial vehicle positioned in an important position in formation flight, the impact on the depiction made by the multiple unmanned aerial vehicles can be reduced.

[0015] Furthermore, the unmanned aerial vehicle monitoring method and program according to the present invention, like the unmanned aerial vehicle monitoring system according to the present invention, can mitigate the impact on the depiction of multiple unmanned aerial vehicles when an abnormality occurs in the flight state of an unmanned aerial vehicle positioned in a critical position in formation flight. [Brief explanation of the drawing]

[0016] [Figure 1] Figure 1 is a diagram illustrating the overview of the unmanned aerial vehicle monitoring system according to this embodiment. [Figure 2] Figure 2 is an example of an external perspective view of an unmanned aerial vehicle (UAV) that is monitored by the UAV monitoring system according to this embodiment. [Figure 3] Figure 3 is a diagram illustrating an example of the hardware configuration of an unmanned aerial vehicle according to this embodiment. [Figure 4]FIG. 4 is a block diagram showing an example of the hardware configuration of the unmanned aerial vehicle monitoring system according to the present embodiment. [Figure 5] FIG. 5 is a block diagram showing an example of the functional configuration of the unmanned aerial vehicle monitoring system according to the present embodiment. [Figure 6] FIG. 6 is an example of a flowchart of the unmanned aerial vehicle monitoring program according to the present embodiment. [Figure 7] FIG. 7 is an example of a flowchart of the unmanned aerial vehicle monitoring program according to another embodiment.

Embodiment for Carrying Out the Invention

[0017] (Overview of Unmanned Aerial Vehicle Monitoring System 10) Referring to FIGS. 1 to 5, an embodiment of the unmanned aerial vehicle monitoring system 10 according to the present disclosure will be described. First, referring to FIG. 1, an overview of the unmanned aerial vehicle monitoring system 10 according to the present embodiment will be described. FIG. 1 is a diagram for explaining the overview of the unmanned aerial vehicle monitoring system 10 according to the present embodiment.

[0018] The unmanned aerial vehicle monitoring system 10 is a so-called information processing device (computer), which is realized by a general-purpose machine such as a server, a personal computer (hereinafter referred to as a PC), a notebook PC, or a tablet PC. Further, the unmanned aerial vehicle monitoring system 10 may be constructed as a dedicated machine for monitoring a plurality of unmanned aerial vehicles 11. The unmanned aerial vehicle monitoring system 10 can remotely operate a plurality of unmanned aerial vehicles 1 with wireless communication. The wireless communication between the unmanned aerial vehicle monitoring system 10 and the unmanned aerial vehicle 11 is performed via the wireless communication antenna 5 shown in FIG. 1 connected to the unmanned aerial vehicle monitoring system 10. Note that the unmanned aerial vehicle monitoring system 10 may be connected to the wireless communication antenna 5 via the information communication network 17. The unmanned aerial vehicle monitoring system 10 enables formation flight of a plurality of unmanned aerial vehicles 11 by remotely operating the plurality of unmanned aerial vehicles 11, and is used for a drone show or the like. When the unmanned aircraft monitoring system 10 is used in a drone show, as shown in FIG. 1, a heart-shaped figure 1 can be depicted in the sky by remotely operating a plurality of unmanned aircraft 11 to fly in formation. As shown in FIG. 1, the unmanned aircraft monitoring system 10 monitors the flight states of each unmanned aircraft 11 during formation flight composed of a plurality of unmanned aircraft 11, especially when depicting figure 1, and responds promptly when an abnormality occurs.

[0019] The unmanned aircraft 11 is equipped with a GPS receiver 31 (see FIG. 3) described later, and measures its own position by receiving GPS signals 4a from an RTK (Real-time kinematic) reference station 6 and GPS (Global Positioning System) satellites 4b to obtain its own position information. This position information and telemetry information on the flight state are transmitted to the unmanned aircraft monitoring system 10 by wireless communication. Real-time kinematic refers to a technology that improves the accuracy of position information by taking into account correction position information of a unique reference station 6 provided on the ground in addition to the position information obtained from the GPS system.

[0020] The unmanned aircraft monitoring system 10 identifies the individuals of the unmanned aircraft 11 based on the individual identification information assigned to the plurality of unmanned aircraft 11. The unmanned aircraft monitoring system 10 has a function of identifying a normal unmanned aircraft 3 and an unmanned aircraft 2 indicating an abnormality based on the individual identification information, and taking prompt measures when an abnormality occurs. Specifically, as shown in FIG. 1(b), when the unmanned aircraft 2 indicating an abnormality is arranged in an important position of the formation, a command to replace the unmanned aircraft 2 with a normal unmanned aircraft 3 that is not in the important position is transmitted to maintain the shape and flight performance of the entire formation. Further, when the position 7 where the unmanned aircraft 11 is missing is detected, an instruction is given to quickly fill the position 7 with a normal unmanned aircraft 11 waiting as a spare to ensure a normal depiction 1. In this way, the unmanned aerial vehicle monitoring system 10 monitors the status of the unmanned aerial vehicles 11 in formation flight in real time, and when an abnormality occurs in an unmanned aerial vehicle 11 in a critical position, it automatically sends a command to replace it with an unmanned aerial vehicle 11 that is not in a critical position and is functioning normally. This reduces the impact on the drawing 1 created by multiple unmanned aerial vehicles 11 when an abnormality occurs in the flight status of an unmanned aerial vehicle 11 positioned in a critical position in formation flight.

[0021] The unmanned aerial vehicle monitoring system 10 processes telemetry information transmitted from each of multiple unmanned aerial vehicles, monitors the flight status of the formation, and has the function of outputting control commands to quickly respond in the event of an anomaly. The unmanned aerial vehicle monitoring system 10 is also intended for use in drone shows in particular.

[0022] Here, a drone show refers to a show event in which multiple unmanned aerial vehicles 11 fly in formation, following a pre-set flight path, and creating letters and shapes in space through lighting and movement effects. By having multiple unmanned aerial vehicles 11 fly in formation, letters, shapes, and dynamic light effects can be expressed in the air, allowing the audience to see the depictions 1 drawn in the air.

[0023] The unmanned aerial vehicle (UAV), also known as a drone, multicopter, or small helicopter, is an aircraft that can be remotely controlled or fly autonomously. More specifically, the UAV 11 is an aircraft whose flight path and movements are controlled based on commands transmitted via radio communication from a ground operator or a pre-implemented program, and which flies at altitude, speed, position, etc., according to the commands transmitted via radio communication or the program. The UAV 11 is equipped with a communication unit 22 (see Figure 3), described later, which transmits telemetry information and its own position information to the UAV monitoring system 10 in real time. Based on this information, the UAV monitoring system 10 can monitor the flight status of the UAV 11 and detect any abnormalities. Specifically, the unmanned aerial vehicle 11 refers to a small quadcopter (an aircraft equipped with four motors), but the number of motors is not limited to this; it may also be an aircraft equipped with six, eight, or other motors. Each unmanned aerial vehicle 11 is equipped with a light-emitting unit 21 (see Figure 2), such as an LED, and displays an image in the air by emitting light from the light-emitting unit 21 at appropriate times.

[0024] Formation flight refers to the simultaneous flight of multiple unmanned aircraft 11 while maintaining coordination with one another. Description 1 is created by the visual performance generated by formation flight.

[0025] An important position is a position in a formation flight where the unmanned aircraft 11 is positioned, and is important compared to other positions. The location of the important position changes as appropriate depending on the shape of the drawing 1, such as letters or figures, drawn by the formation flight. For example, a position on the outline of a figure or letters drawn by the formation flight of multiple unmanned aircraft 11 may be considered an important position, and a position inside the outline of the figure or letters may not be considered an important position. Furthermore, the position in which the unmanned aircraft 11 is placed may be determined by the luminance of the unmanned aircraft 11 to determine whether or not it is a critical position. For example, if the luminance of the radio-controlled aircraft is set to a predetermined value or higher, the position in which the unmanned aircraft 11 is placed may be considered a critical position, and if the luminance of the unmanned aircraft 11 is set to a value lower than the predetermined value, the position in which the unmanned aircraft 11 is placed may not be considered a critical position. Furthermore, positions that are easily visible to the audience may be designated as important positions, while positions that are difficult to see from the audience may not be designated as important positions. Furthermore, the central position of the depiction 1 drawn by multiple unmanned aircraft 11 may be considered an important position, while positions other than the central position may not be considered important positions. An abnormal flight status refers to the determination made by the unmanned aerial vehicle monitoring system 10 regarding the flight status of the unmanned aerial vehicle 11 that is the source of the telemetry information, when the telemetry information contains information indicating an anomaly. Specifically, an abnormal flight status means that the flight status of the unmanned aerial vehicle 11 (e.g., speed, altitude, position, luminescence of the light-emitting unit 21, temperature of the main unit 18, etc.) falls outside a preset range. An abnormal flight status also includes a state in which the battery level of the battery 27, described later, falls below a preset threshold.

[0026] Flight status refers to the state in which an unmanned aerial vehicle 11 or aircraft is in flight, comprehensively indicating various flight-related information such as its operational status, position, altitude, and speed. In detail, the flight status includes parameters such as altitude, attitude, speed, position coordinates, battery level 27, and temperature that the unmanned aerial vehicle 11 should maintain during flight. Based on this information, it is possible to maintain normal flight of the unmanned aerial vehicle 11 and detect abnormalities. Flight status is generally monitored and acquired in real time and plays an important role in flight safety and mission accomplishment. For example, flight status includes altitude acquired by the altitude sensor 34, attitude information from the gyro sensor, acquisition of position coordinates using GPS signal 4a, and display of remaining battery level through battery monitoring. This allows monitoring of flight status to confirm whether the unmanned aircraft 11 in formation flight are maintaining their planned flight path and altitude. The replacement command is a command to replace unmanned aerial vehicle 2, which is located in a critical position and exhibiting abnormalities, with unmanned aerial vehicle 3, which is located in a non-critical position and is functioning normally (not exhibiting abnormalities).

[0027] (Regarding the configuration of the unmanned aerial vehicle 11) Next, the configuration of the unmanned aerial vehicle 11 will be described with reference to Figures 2 and 3. Figure 2 is an example of an external perspective view of the unmanned aerial vehicle 11 that is monitored by the unmanned aerial vehicle monitoring system 10, and Figure 3 is a diagram illustrating an example of the hardware configuration of the unmanned aerial vehicle 11. As shown in Figure 2, the unmanned aerial vehicle 11 has a main body 18 in the center and four arms 19 (first arm 19a, second arm 19b, third arm 19c, and fourth arm 19d) extending in all directions from the main body 18. The tips of the four arms 19 are equipped with propellers 20. The propellers 20 are rotationally driven by a flight drive mechanism 30 (see Figure 3) which consists of a motor and the like. Specifically, the tip of the first arm 19a is equipped with a first propeller 20a, the tip of the second arm 19b is equipped with a second propeller 20b, the tip of the third arm 19c is equipped with a third propeller 20c, and the tip of the fourth arm 19d is equipped with a fourth propeller 20d. The propeller 20 shown in Figure 2 is a two-bladed propeller, but it may also be a three-bladed or four-bladed propeller, etc.

[0028] As shown in Figure 3, the unmanned aerial vehicle 11 includes a communication unit 22, a ROM (ReadOnly Memory) 23, a RAM (RandomAccess Memory) 24, a storage unit 25, a control unit 26, a battery 27, an input / output interface 28, and the like. Furthermore, the unmanned aerial vehicle 11 is equipped with a flight drive unit 29, a flight drive mechanism 30, a GPS receiver 31, an illuminance sensor 32, a geomagnetic sensor 33, an altitude sensor 34, a gyro sensor 35, an obstacle detection camera 36, ​​and a light-emitting unit (LED) 21, which are connected to the control unit 26 via an input / output interface 28 to enable bidirectional data communication.

[0029] The communication unit 22 is primarily used for wireless communication between the unmanned aerial vehicle monitoring system 10 and other unmanned aerial vehicles 11. The communication method of the communication unit 22 is not particularly limited and may include Wi-Fi®, LoRa®, Bluetooth®, Zigbee®, infrared wireless communication, microwave wireless communication, broadcast wireless communication, satellite communication, etc. The communication unit 22 may also perform wireless communication between the unmanned aerial vehicle monitoring system 10 and other unmanned aerial vehicles 11 via the wireless antenna 12.

[0030] The ROM (Read Only Memory) 23 can be used as a recording device and stores firmware, various applications, and various data used for controlling the operation of each functional part of the unmanned aerial vehicle 11. The RAM (Random Access Memory) 24 is used to configure the main memory accessed by the control unit 26, and may also be used to temporarily store data acquired from various sensors such as the GPS receiver 31 and the illuminance sensor 32 mounted on the unmanned aerial vehicle 11. The memory unit 25 is a recording medium such as an SD card or USB memory, or a storage device such as an SSD (Solid State Drive), and stores the light emission pattern information and flight pattern information described later. Alternatively, the memory unit 25 may be used to temporarily store data acquired from various sensors such as the GPS receiver 31 and the illuminance sensor 32, instead of the RAM 24. Furthermore, the memory unit 25 may not be mounted on the unmanned aerial vehicle 11, and the RAM 24 may perform the function of the memory unit 25.

[0031] The control unit 26 is mainly used to execute the firmware of the unmanned aerial vehicle 11 and includes a central processing unit (CPU), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), etc., and is realized by logic circuits (hardware) formed by integrated circuits (IC (Integrated Circuit) chips, LSI (Large Scale Integration)), etc., and dedicated circuits. The battery 27 serves as the power source for the unmanned aerial vehicle 11 and supplies power to various functional units. For example, a lithium polymer battery is used for the battery 27. It is desirable that the battery 27 be used with an overcharge protection function to protect it from overcharging. In addition, the battery 27's remaining charge is monitored by a battery charge check unit (not shown), and if the battery 27's remaining charge falls below a predetermined value, the user is notified of the low battery level.

[0032] The input / output interface 28 is an interface for sending and receiving data to and from the flight drive unit 29, flight drive mechanism 30, GPS receiver 31, illuminance sensor 32, geomagnetic sensor 33, altitude sensor 34, gyro sensor 35, obstacle detection camera 36, ​​and light-emitting unit (LED) 21, etc. The input / output interface 28 uses different standards depending on the data being handled, and may support multiple standards. Examples of standards include USB 2.0, USB 3.0, RS-232C, IEEE 1394, SCSI, and SASI. The flight drive unit 29 controls the flight drive mechanism 30 based on the flight drive signal generated by the control unit 26. The flight drive mechanism 30 is a mechanism, consisting of a motor and the like, for rotating the propeller 20.

[0033] The GPS (Global Positioning System) receiver 31 acquires positional information of the unmanned aerial vehicle 11, including its latitude, longitude, and altitude. The acquired positional information of the unmanned aerial vehicle 11 is stored in the RAM 24 or memory unit 25. To obtain more accurate positional information, the current position may be corrected using radio waves from a mobile phone base station or radio waves from a wireless communication access point (Wi-Fi®, Bluetooth®, etc.). Alternatively, the accuracy of the positional information of the unmanned aerial vehicle 11 may be improved by using a quasi-zenith satellite system or RTK-GPS (Real Time Kinematic - Global Positioning System). The Quasi-Zenith Satellite System (QZSS) is a satellite positioning system provided by Japan. Like GPS, it provides location information, but with higher accuracy. In particular, it improves positioning accuracy in certain regions of Japan. RTK-GPS improves the accuracy of positional information by incorporating corrected positional information from electronic reference points installed on the ground in addition to positional information obtained from GPS. While the typical positional error of positional information obtained from GPS is said to be around a few meters, the positional error of positional information obtained from RTK-GPS is said to be a few centimeters. In drone shows, formation flights are performed by multiple unmanned aerial vehicles 11, so each unmanned aerial vehicle 11 needs to be positioned in the air with high precision, and RTK-GPS may be used for this purpose.

[0034] The illuminance sensor 32 measures the illuminance outside the unmanned aerial vehicle 11. The illuminance sensor 32 is mounted on each unmanned aircraft 11 and measures the external illuminance for each unmanned aircraft 11. The measurement results obtained by the illuminance sensor 32 are used by the luminous luminance correction unit 26b, which will be described later. The luminous intensity correction unit 26b corrects the luminous intensity included in the light emission pattern information of the light emission unit 21 based on the measurement results of the illuminance sensor 32. The luminescence correction unit 26b corrects the luminescence of the light-emitting unit 21 according to the external illumination for each unmanned aircraft 11, thereby suppressing insufficient brightness of the light-emitting unit 21 when displaying images. Since the light-emitting unit 21 consumes a lot of power, the brightness of the light-emitting unit 21 can be appropriately corrected by the light-emitting brightness correction unit 26b, thereby suppressing any adverse effects on the battery capacity and flight time of the battery 27.

[0035] The geomagnetic sensor 33 is used to detect the direction in which the unmanned aerial vehicle 11 is flying. The altitude sensor 34 is used to detect the current altitude of the unmanned aerial vehicle 11 and includes, for example, a barometric pressure sensor. The gyro sensor 35 detects changes in acceleration in the unmanned aerial vehicle 11 and is used to detect the direction of travel, speed, and attitude of the unmanned aerial vehicle 11. The obstacle detection camera 36 is used to detect objects that would obstruct the flight movement of the unmanned aerial vehicle 11, and is used, for example, to prevent collisions with other unmanned aerial vehicles 11 and to detect landing sites for the unmanned aerial vehicle 11. The light-emitting unit 21 is used to illuminate the unmanned aerial vehicle 11, and mainly LEDs are used. The LED may be a single-color LED or a multi-color LED. A multi-color LED is one that produces a variety of colors by mixing three colors of light, RGB (red, green, blue) or four colors of light, RGBW (red, green, blue, white). The light-emitting unit 21 may also emit laser light. The LEDs used in the light-emitting section 21 should ideally be lightweight, highly durable, and high-brightness.

[0036] The control unit 26 loads the firmware stored in the ROM 23 into the main memory, which is composed of RAM 24 and the like. The control unit 26 accesses the main memory into which the firmware has been loaded and executes the firmware. By executing the firmware, the control unit 26 provides functional units such as the receiving unit 26a and the light emission brightness correction unit 26b.

[0037] The receiving unit 26a receives position information and light emission pattern information transmitted from the transmitting unit 47 of the unmanned aircraft monitoring system 10, as described later, during flight. Position information refers to the position information of each dot of the unmanned aerial vehicle 11 that constitutes the matrix display of the image shown in the air, and is expressed in (x, y, z) coordinates in a three-dimensional Cartesian coordinate system. In addition to the three-dimensional Cartesian coordinate system, the position information may also be expressed using a spherical coordinate system, a cylindrical coordinate system, or a polar coordinate system. The position information is unique to each dot that makes up the matrix display of the image shown in the air, and is unique to the unmanned aerial vehicle 11 assigned to each dot. The receiving unit 26a receives the position information corresponding to the dot assigned to the unmanned aerial vehicle 11 of the receiving unit 26a. Light emission pattern information refers to control information that defines the light emission pattern for each dot that constitutes the matrix display of an image shown in the air, and controls the light emission unit 21 of the unmanned aircraft 11 assigned to each dot. The light emission mode refers to the light emission color, light emission brightness, and light emission timing of the light emission unit 21. When an LED is used in the light emission unit 21, the light emission mode refers to the light emission color, light emission brightness, and light emission timing of the LED.

[0038] As described above, the luminescence brightness correction unit 26b corrects the luminescence brightness included in the luminescence pattern information based on the measurement results of the illuminance sensor 32. Luminous brightness refers to the luminous brightness of the light-emitting unit 21, and if an LED is used in the light-emitting unit 21, it refers to the luminous brightness of the LED.

[0039] (Regarding the hardware configuration of the unmanned aerial vehicle monitoring system 10) Next, with reference to Figure 4, an example of the hardware configuration of the unmanned aerial vehicle monitoring system 10 will be described. Figure 4 is a block diagram illustrating an example of the hardware configuration of the unmanned aerial vehicle monitoring system 10. The unmanned aerial vehicle monitoring system 10 includes a communication unit 10a, ROM 10b, RAM 10c, storage unit 10d, processing unit 10e, and input / output interface 10f, among others. Furthermore, the unmanned aerial vehicle monitoring system 10 includes a display unit 10g, an operation input unit 10h, and an external information input unit 10i, which perform data input and output via an input / output interface 10f.

[0040] The communication unit 10a is equipped with the function of performing bidirectional communication between the unmanned aerial vehicle 11 and other information processing equipment. When the communication unit 10a performs bidirectional communication with the unmanned aerial vehicle 11, it does so by wireless communication, which may be done via the wireless antenna 12 (see Figure 1), or via a wireless antenna (not shown) connected to the information and communication network 17. The wireless communication method is not particularly limited and may include Wi-Fi®, LoRa®, Bluetooth®, Zigbee®, infrared wireless communication, microwave wireless communication, broadcast wireless communication, satellite communication, etc. The other information processing device may be another unmanned aerial vehicle monitoring system 10, or any other information processing device. When the communication unit 10a performs bidirectional communication with other information processing devices, it may do so via the information communication network 17, or it may connect directly to the other information processing devices to perform bidirectional communication. The communication unit 10a may use wired communication or wireless communication for communication with other information processing devices. The external information input unit 10i, described later, may receive original image data from another information processing device via the communication unit 10a.

[0041] ROM10b can be used as a recording device and stores the BIOS necessary for controlling the operation of each functional part of the unmanned aerial vehicle monitoring system 10, as well as various data used in the BIOS. BIOS is a program that manages the basic input / output functions of the unmanned aerial vehicle monitoring system 10 as an information processing device. It is the first program to run when the unmanned aerial vehicle monitoring system 10 is powered on, and it controls hardware such as the communication unit 10a, ROM 10b, RAM 10c, storage unit 10d, processing unit 10e, and input / output interface 10f, and prepares the OS (Operating System) to start up.

[0042] RAM10c is used to configure the main memory accessed by the processing unit 10e, and is also used to temporarily store various data acquired or generated by the unmanned aerial vehicle monitoring system 10 before storing them in the storage unit 10d.

[0043] The storage unit 10d is implemented using an HDD (Hard Disk Drive), SSD (Solid State Drive), online storage, etc., and stores the OS, other application software described later, and various data used by these programs. The storage unit 10d also stores various data acquired or generated by the unmanned aerial vehicle monitoring system 10.

[0044] The processing unit 10e includes a CPU, MPU, GPU, etc., and is implemented by logic circuits and dedicated circuits formed by integrated circuits (IC chips, LSIs), etc. The input / output interface 10f is an interface for sending and receiving data to and from the display unit 10g, the operation input unit 10h, and the external information input unit 10i, etc. The input / output interface 10f uses different standards depending on the data being handled, and may support multiple standards. Examples of standards include HDMI (registered trademark), USB 2.0, USB 3.0, RS-232C, IEEE 1394, SCSI, and SASI.

[0045] The display unit 10g is a monitor for the unmanned aerial vehicle monitoring system 10 and displays various data. The operation input unit 10h receives input from the user regarding the operation of the unmanned aerial vehicle monitoring system 10. The operation is performed using a keyboard, mouse, etc., and the output signals from the keyboard, mouse, etc. are input to the operation input unit 10h. The external information input unit 10i receives input from an external source of raw image data that will be used as the basis for the images displayed by the unmanned aerial vehicle monitoring system 10. The original image data is the image data that serves as the source material for the image displayed by the unmanned aerial vehicle monitoring system 10, and an image represented by a dot matrix display is obtained based on the original image data. Furthermore, the user of the unmanned aerial vehicle monitoring system 10 may use this original image data to derive an image that they wish to display in the air using multiple unmanned aerial vehicles 11. The source image data may be a still image, a video, or it may be text, music, symbols, a code, etc. The external information input unit 10i receives this source image data and inputs it to the unmanned aerial vehicle monitoring system 10. The external information input unit 10i may be connected to external input devices such as cameras, video cameras, microphones, and scanners, and may accept output data from these input devices as input data.

[0046] If the image displayed in the air by the unmanned aerial vehicle monitoring system 10 is, for example, an animation in a drone show, the animation is created by following the procedure below. (Step 1) First, an animation is created based on the original image data. To create the animation, you must decide on the people and objects that will appear, as well as the backgrounds to be used. People include people, animals, mascots, etc. Objects include vehicles, buildings, supplies, tools, etc., as well as text, illustrations, marks, codes, etc. Codes may include QR codes (registered trademarks), barcodes, codes, etc. Backgrounds include, for example, mountains, forests, rivers, night skies, snowscapes, fireworks, or landscapes. In animation, once the characters and objects that will appear are decided, the next step is to determine how they will move.

[0047] (Step 2) Next, we will determine the types and number of unmanned aerial vehicles (UAVs) 11. Determine the 11 types and number of unmanned aerial vehicles to be used to create the animation. Once the animation is created, it is converted to a dot matrix display to determine the number of dots required to display the animation. The determined number of dots will be the number of unmanned aerial vehicles (UAVs) 11 to be used. The type of unmanned aerial vehicle 11 is determined by factors such as its possible continuous flight time and the number of emission colors. The possible continuous flight time of the unmanned aerial vehicle 11 must be at least longer than the animation's running time. The more emission colors the unmanned aerial vehicle 11 can emit, the more diverse the animation can be.

[0048] (Step 3) Create flight pattern information for the unmanned aerial vehicle 11. Flight pattern information is created that defines the flight route, flight timing, and flight speed of each unmanned aerial vehicle (UAV) 11 assigned to each dot in the dot matrix display that makes up the animation. The flight pattern information is different for each dot in the dot matrix display. Multiple UAVs 11 fly according to their respective flight pattern information to create a specific pattern or design. The animation is realized when multiple UAVs 11 fly simultaneously and synchronize in the air. Furthermore, if the animation does not involve movement, and therefore the image displayed in the air by illuminating multiple unmanned aerial vehicles 11 is a still image, the flight pattern information will include the flight route to the position information of each dot that constitutes the dot matrix display of the still image. In addition, the flight pattern information may or may not include the return route of the unmanned aerial vehicle 11. If the return route is not included in the flight pattern information, the unmanned aerial vehicle 11 will autonomously select a flight route towards a predetermined return location while confirming its current position using GPS signals 4a.

[0049] (Step 4) Create light pattern information for the unmanned aerial vehicle 11. Based on the colors of the original image data, light emission pattern information is created that defines the light emission mode of the unmanned aerial vehicle 11 assigned to each dot of the dot matrix display used to display the animation. The light emission mode refers to the light emission color, light emission brightness, and light emission timing of the LEDs of the light emission section 21 of the unmanned aerial vehicle 11. The LED color and blinking pattern are set to draw designs such as characters, codes, and shapes.

[0050] (Regarding the functional configuration of the unmanned aerial vehicle monitoring system 10) Next, with reference to Figure 5, an example of the functional configuration of the unmanned aerial vehicle monitoring system 10 will be described. Figure 5 is a block diagram illustrating an example of the functional configuration of the unmanned aerial vehicle monitoring system 10. The unmanned aerial vehicle monitoring system 10 monitors the flight status of multiple unmanned aerial vehicles 3 in formation flight. The unmanned aerial vehicle monitoring system 10 loads the unmanned aerial vehicle monitoring program, described later, stored in the memory unit 10d, into the main memory, which is composed of RAM 10c and the like. The processing unit 10e accesses the main memory into which the unmanned aerial vehicle monitoring program has been loaded and executes the unmanned aerial vehicle monitoring program. The unmanned aerial vehicle monitoring system 10, by executing the unmanned aerial vehicle monitoring program, has a processing unit 10e equipped with functional units such as a telemetry information acquisition unit 40, an anomaly detection unit 41, a captured image acquisition unit 42, a learning model acquisition unit 43, a depiction anomaly detection unit 44, an anomaly extraction unit 45, a critical position determination unit 46, and a transmission unit 47.

[0051] (Telemetry information acquisition unit 40) The telemetry information acquisition unit 40 acquires telemetry information indicating the flight status of the unmanned aircraft 11 during formation flight, which is transmitted from each of the unmanned aircraft 11 and associated with individual identification information that is pre-assigned to each of the multiple unmanned aircraft 11 and identifies each individual unmanned aircraft 11. In detail, the telemetry information acquisition unit 40 is responsible for acquiring telemetry information such as altitude, temperature, battery level, attitude, and position coordinates transmitted from each unmanned aerial vehicle 11 in real time, based on the individual identification information assigned to each unmanned aerial vehicle 11. This telemetry information is important data for detecting anomalies in the unmanned aerial vehicles 11 and for maintaining formation. For example, the telemetry information acquisition unit 40 uses GPS signals 4a and data from the IMU (Inertial Measurement Unit) to acquire the position coordinates and attitude of the unmanned aircraft, and to accurately confirm its position within the formation. Real-time acquisition of such telemetry information allows for the rapid replacement or correction of the aircraft in the event of an anomaly, ensuring the safety and efficiency of the formation. Individual identification information is information that is assigned to each individual unmanned aerial vehicle 11 in advance, and is used to identify each unmanned aerial vehicle 11 individually. For example, individual identification information may be represented by numbers, or a combination of numbers and letters.

[0052] (Abnormality determination unit 41) The abnormality detection unit 41 determines whether there is an abnormality in the flight state of the unmanned aircraft 11 based on the telemetry information. The abnormality detection unit 41 analyzes the telemetry information acquired from the unmanned aircraft 11, such as altitude, temperature, battery level, attitude, and position coordinates, and determines whether there is an abnormality by checking whether this information is within a preset normal range. For example, the abnormality detection unit 41 detects abnormalities based on telemetry information, such as the unmanned aircraft 11 deviating from its set flight altitude or the battery level decreasing.

[0053] (Image acquisition unit 42) The image acquisition unit 42 is an image acquisition unit 42 that acquires images captured by a camera that captures the depiction drawn by multiple unmanned aerial vehicles during formation flight. The image acquisition unit 42 of the present invention acquires the flight status of the formation as image data in real time via a camera that captures the shape and positional relationship of the formation drawn by the unmanned aerial vehicles. These captured images are used as an important source of information for the subsequent depiction abnormality determination unit 44 to determine whether or not there is an abnormality. For example, the image acquisition unit 42 can use a wide-angle lens camera or a high-resolution camera to visually detect abnormalities such as misalignment or deformation of the unmanned aircraft 11 during formation flight. In this way, the image acquisition unit 42 understands the flight status of the unmanned aircraft 11 and provides important data to ensure the safety and stability of the formation.

[0054] (Learning model acquisition unit 43) The learning model acquisition unit 43 further includes a learning model that has previously learned normal depictions 1, determines whether or not there are any abnormalities in the depictions included in the captured image, and, if it is determined that there are abnormalities in the depictions, acquires a learning model that outputs individual identification information of the unmanned aerial vehicle that caused the abnormality. Specifically, the learning model acquisition unit 43 works in conjunction with the processing unit 10e to pre-learn features such as contour shape and brightness in a normal formation depiction 1 using machine learning. The model obtained by the learning model acquisition unit 43 based on this learning model acquisition step 43a is used when analyzing image data acquired from the captured image acquisition unit 42, and in the process of the depiction anomaly detection step 44a performed by the depiction anomaly detection unit 44, depictions that deviate from the normal range and distortions in shape are detected. If an anomaly is detected, the anomaly extraction unit 45 executes the anomaly extraction step 45a and, while comparing it with a normal depiction previously learned by the learning model acquisition unit 43, quickly extracts the individual identification information of the UAV 11 causing the anomaly. For example, if an UAV 2 exhibiting an anomaly in the formation deviates from its normal position, by identifying its individual identification information, the transmission unit 47 of the UAV monitoring system 10 sends an appropriate correction command to the UAV 11 in question and issues a command to move an UAV 3 that does not exhibit an anomaly to the missing position 7. The learning model pre-learns the shape and brightness of the contours of normal depictions, determines whether there are any abnormalities in depiction 1 included in the captured image, and outputs individual identification information of the unmanned aerial vehicle 11 that caused the abnormality if it is determined that there are abnormalities in depiction 1 included in the captured image. The learning model is characterized by pre-learning the shape and brightness of the contours of normal depictions, determining whether there are any abnormalities in the depictions included in the captured image, and outputting individual identification information of the unmanned aerial vehicle 11 that is causing the abnormality if it is determined that there are abnormalities in the depictions included in the captured image.

[0055] (Description abnormality determination unit 44) The depiction abnormality determination unit 44 determines whether or not there is an abnormality in depiction 1 based on the captured image. Specifically, the depiction anomaly detection unit 44 compares the contour shape and brightness information of a normal depiction 1, which has been acquired in advance by the learning model acquisition unit 43, with the actual depiction based on the captured image, and makes an anomaly determination. For example, if the unmanned aircraft 2 that is showing an anomaly is deviated from its normal position or its brightness is reduced, the depiction anomaly detection unit 44 detects this deviation and change in brightness and determines that it is an anomaly. When an anomaly is detected, the anomaly extraction unit 45 executes the anomaly extraction step 45a to identify the individual identification information of the unmanned aerial vehicle 2 that caused the anomaly. This individual identification information is transmitted to the transmission unit 47, and if the unmanned aerial vehicle 2 that has experienced an anomaly is located in an important position 7, a movement command is sent for a normal unmanned aerial vehicle 3 to fill that position. Thus, the depiction anomaly detection unit 44 has the effect of accurately detecting anomalies in the shape and brightness of the depiction 1 drawn by the unmanned aircraft 11 during formation flight, enabling a rapid response to the anomaly.

[0056] (Abnormal extraction part 45) The anomaly extraction unit 45 extracts individual identification information of the unmanned aircraft 11 that caused the anomaly when it is determined that depiction 1 is an anomaly. Specifically, the anomaly detection unit 45 compares the contour and brightness data of a normal depiction 1 constructed by the learning model acquisition unit 43, and when an unmanned aerial vehicle 2 with an abnormal position and brightness is identified, it quickly extracts the individual identification information of that unmanned aerial vehicle 11. This individual identification information is transmitted to the transmission unit 47, and a command is sent to move a normal unmanned aerial vehicle 3 to the location where the anomaly occurred, thereby ensuring the stability of the formation flight shape and position. Because the anomaly detection unit 45 quickly identifies the cause of an anomaly, even if an anomaly occurs in the formation flight of the unmanned aircraft 11, the impact is minimized. Furthermore, in visual performances such as drone shows, it is possible to provide the audience with a stable and uninterrupted performance. The depiction anomaly detection unit 44 uses a learning model to determine whether there are any anomalies in the depiction 1 included in the captured image, and the anomaly extraction unit 45 uses a learning model to extract individual identification information of the unmanned aerial vehicle 11 that is the cause of the anomaly included in the captured image. Specifically, when the depiction anomaly detection unit 44 detects an anomaly in the position or brightness of depiction 1, the anomaly extraction unit 45 executes an anomaly extraction step 45a and transmits the individual identification information of the unmanned aerial vehicle 11 identified as the cause of the anomaly to the transmission unit 47. The transmission unit 47 issues a command to replace the unmanned aerial vehicle 2 that has experienced an anomaly with a normal unmanned aerial vehicle 3, thereby stabilizing the formation shape and flight performance. For example, when an anomaly is detected, the anomaly detection unit 45 uses a learning model and telemetry information to extract individual identification information for a specific unmanned aerial vehicle (UAV) 11 if the anomaly relates to the position or attitude of that UAV. This information is used to issue a command to replace the UAV in the anomaly position with a normal one, and functions as a foundation for ensuring the stability of the formation.

[0057] (Important position determination unit 46) The important position determination unit 46 determines whether the unmanned aircraft 11 is positioned in an important position in the formation, based on the individual identification information which includes position information relating to the position of the unmanned aircraft 11 in the formation. Specifically, the telemetry information acquisition unit 40 acquires telemetry information on the flight status transmitted from each unmanned aircraft 11, and the abnormality determination unit 41 checks for abnormalities based on this information. If an abnormality is detected in the flight status of an unmanned aircraft 11, the critical position determination unit 46 determines whether the unmanned aircraft 11 experiencing the abnormality is in a critical position 7 necessary to maintain the formation. For example, if it is determined that an abnormal unmanned aircraft 2 is positioned in a critical position 7 within the formation, the transmitter 47 sends a command to a normal unmanned aircraft 3 that is not experiencing any abnormalities, and moves the normal unmanned aircraft 3 to the position of the abnormal unmanned aircraft 2, thereby stabilizing the formation's shape and flight performance. Through this series of actions, even if an abnormality occurs in the positioning of the unmanned aircraft 11, the consistency and stability of the formation are ensured. With this configuration, the critical position determination unit 46 can always appropriately position the unmanned aircraft 11 at key positions in the formation and maintenance of the formation, and has the effect of performing rapid corrections in the event of an anomaly.

[0058] (Transmitter 47) The transmitting unit 47 transmits a command to the radio-controlled aircraft in formation flight to replace an unmanned aircraft 11 that has been determined to be in an important position and is found to have an abnormality with an unmanned aircraft 11 that has been determined not to be in an important position and is found to have no abnormality. Specifically, based on the data collected from the telemetry information acquisition unit 40, when the anomaly detection unit 41 detects an anomaly in the unmanned aerial vehicle 11, the critical position determination unit 46 checks whether the anomalying unmanned aerial vehicle 11 is located in a critical position 7. Depending on the result, the transmission unit 47 instructs the unmanned aerial vehicle 3, which is not experiencing an anomaly, to move to the location of the unmanned aerial vehicle 2 where the anomaly occurred. For example, when an unmanned aircraft 2 exhibiting an anomaly is in a critical position 7, the transmitter 47 issues a command for a normal unmanned aircraft 3 to move to that position, immediately correcting any disruption to the formation caused by the anomaly. This configuration ensures the stability and safety of the formation in the event of an anomaly.

[0059] The important position determination unit 46 determines whether the unmanned aircraft 11 related to the individual identification information extracted by the anomaly extraction unit 45 is positioned in an important position. If the transmission unit 47 determines that the unmanned aircraft 11 related to the individual identification information extracted by the anomaly extraction unit 45 is positioned in an important position, it transmits a command to the radio aircraft in formation flight to replace the unmanned aircraft 11 with an unmanned aircraft 11 that has been determined not to be positioned in an important position and is determined to be free of anomalies. Specifically, after analyzing the image data acquired by the image acquisition unit 42, the depiction anomaly presence / absence determination unit 44 detects an anomaly in depiction 1, and the anomaly extraction unit 45 identifies the individual identification information of the unmanned aircraft 11 causing the anomaly. Then, the important position determination unit 46 determines whether the unmanned aircraft 11 is positioned in an important position 7. If it is determined that the malfunctioning unmanned aircraft 2 is in critical position 7, the transmitter 47 will send a command to the normal unmanned aircraft 3 to move to the position of the malfunctioning unmanned aircraft 2. This command allows a normal drone 3 to quickly replace the malfunctioning drone 2, maintaining consistency in the formation's shape and visual representation 1. This minimizes disruption to the formation caused by the malfunction, ensuring a stable visual performance for audiences at events such as drone shows. For example, if the abnormality detection unit 41 detects an abnormality and the critical position determination unit 46 determines that the unmanned aircraft is positioned in a critical position, the transmitter unit 47 sends a command to an unmanned aircraft that is not experiencing an abnormality to move to that position. This function ensures the safety of the formation and prevents abnormalities from affecting the overall flight performance.

[0060] For example, suppose in the unmanned aerial vehicle monitoring system 10, five unmanned aerial vehicles are flying in a V-shaped formation. Among them, the unmanned aerial vehicle at the front of the V-shape is in a critical position. Telemetry information reveals that the battery level of the leading unmanned aerial vehicle has fallen below a predetermined threshold during flight. Upon receiving this information, the monitoring system immediately checks for an anomaly in the flight status through the anomaly detection unit 41, and the critical position detection unit 46 confirms that the leading unmanned aerial vehicle is in a critical position. Subsequently, the transmission unit 47 sends a command to another normal unmanned aerial vehicle in the formation to move to the front position and replace the unmanned aerial vehicle 2 that has experienced an anomaly.

[0061] One possible implementation of this embodiment is to install a monitoring system that collects data from various sensors of the unmanned aerial vehicle 11 in real time and centrally manages and analyzes it. This system constantly updates information such as the position, speed, altitude, and battery status of the unmanned aerial vehicle 11, and has the function of immediately transmitting a response command to the unmanned aerial vehicle if an abnormality is detected. This makes it possible to significantly improve the safety and efficiency of formation flight.

[0062] This embodiment relates to an unmanned aerial vehicle (UAV) monitoring system 10 that enables monitoring of the flight status of UAVs 11 during formation flight and rapid response in the event of an anomaly. In particular, it has the function of determining whether an UAV is positioned in an important position in the formation, and if an anomaly occurs in an UAV positioned in an important position, it can issue a command to replace it with an UAV that is not positioned in that position and is in a normal flight state.

[0063] In this embodiment, the unmanned aerial vehicle (UAV) 11 is equipped with numerous sensors that transmit data from these sensors to the UAV monitoring system 10 in real time. This system monitors the flight status of the UAV 11 based on the received data and has the function of immediately transmitting a response command to the UAV 11 when an anomaly is detected. The system also determines whether each UAV 11 is in a critical position, and if an anomaly is detected in an UAV in a critical position, it quickly replaces it with another UAV 11.

[0064] Thus, the unmanned aerial vehicle monitoring system 10 of this embodiment is an important technology for improving the safety and efficiency of unmanned aerial vehicles during formation flight, and is expected to have applications in a wide range of fields, including disaster investigation, agriculture, and logistics.

[0065] In this embodiment, the unmanned aerial vehicle 11 is equipped with multiple sensors and a flight control device, and collects and monitors flight data in real time during formation flight. In particular, if the battery level falls below a preset threshold, the abnormality detection unit 41 determines that this state is abnormal, and the transmission unit 47 issues a command to replace it with another unmanned aerial vehicle 11. This system also has a function to share power between the unmanned aerial vehicles 11, so that the power supply device of one unmanned aerial vehicle 11 can supply power to the other unmanned aerial vehicle 11. This enables safe and efficient formation flight while extending the flight distance and time.

[0066] The unmanned aerial vehicle monitoring system 10 of this embodiment has a function to reacquire data through other communication channels if a failure occurs in receiving data from the unmanned aerial vehicle 11, by providing redundancy using multiple communication channels. This makes it possible to significantly improve the safety and reliability when the unmanned aerial vehicles 11 perform formation flight. In particular, it is possible to respond quickly even when a communication failure occurs, thus maintaining the efficiency and safety of formation flight.

[0067] This embodiment describes a system in which unmanned aerial vehicles (UAVs) 11 fly in formation, sharing power with each other. Each UAV is equipped with a flight control device, an openable and closable ring body, a current generator, a ring opening and closing device, and a power supply device. The UAVs 11 are electrically connected via cables, allowing power to be supplied from one UAV 11 to another. During formation flight, the UAVs approach overhead power lines and open and close the ring bodies to accommodate the overhead power lines within the rings. The power generated by electromagnetic induction from the overhead power lines is supplied to the thrust generators and flight control devices of the UAVs, enabling efficient and sustainable flight.

[0068] In this embodiment, the unmanned aerial vehicle 11 monitors its flight status in real time, and in particular, if the battery level falls below a preset threshold, the abnormality detection unit 41 determines that this is an abnormality. Based on this information, the transmission unit 47 sends a command to replace the unmanned aerial vehicle 11 in the formation, and the unmanned aerial vehicle 2 that has experienced an abnormality is quickly replaced. This system enables the unmanned aerial vehicle 11 to continue long-distance and long-duration flights safely and efficiently, and can significantly improve the safety and efficiency of formation flight.

[0069] In this embodiment, a system is provided to quickly replace an unmanned aerial vehicle (UAV) 2 with another UAV 3 in a normal state if an abnormality occurs in an UAV 11 positioned at a specific critical location while the UAVs 11 are performing formation flight. Each UAV 11 is equipped with a flight control device, an annular ring body, a current generator, a ring opening / closing device, and a power supply device, and overhead power lines are housed inside the ring body to supply current generated by electromagnetic induction to the flight control device. This system improves the efficiency and safety of formation flight and enhances the efficiency of inspecting overhead power lines.

[0070] In this embodiment, the unmanned aerial vehicle 11 has the ability to reacquire data through other communication channels if a failure occurs in data reception during flight, by utilizing multiple communication channels to provide redundancy. This significantly improves the safety and reliability of the unmanned aerial vehicle 11 when performing formation flight, and in particular, it can respond quickly in the event of a communication failure, thereby maintaining the efficiency and safety of formation flight.

[0071] (Regarding methods and programs for monitoring unmanned aerial vehicles) Next, with reference to Figure 6, an unmanned aerial vehicle (UAV) monitoring program according to one embodiment of the present disclosure will be described together with an UAV monitoring method. Figure 6 is a flowchart of the UAV monitoring program according to this embodiment. The UAV monitoring method is executed by the processing unit 10e of the UAV monitoring system 10 based on the UAV monitoring program.

[0072] The unmanned aerial vehicle monitoring program includes steps such as acquiring telemetry information (S40), determining whether or not there is an anomaly (S41), determining an important position (S46), and transmitting information (S47).

[0073] The unmanned aerial vehicle (UAV) monitoring program causes the processing unit 10e of the UAV monitoring system 10 to perform various functions such as telemetry information acquisition, anomaly detection, critical position detection, and transmission. Since these functions overlap with the descriptions of the various functional units of the UAV monitoring system 10 mentioned above, a detailed explanation will be omitted.

[0074] The telemetry information acquisition function is associated with individual identification information that is pre-assigned to multiple unmanned aircraft 11 and identifies each individual unmanned aircraft 11, and acquires telemetry information indicating the flight status of the unmanned aircraft 11 during formation flight, which is transmitted from each of the unmanned aircraft 11 (telemetry information acquisition step S40).

[0075] The abnormality detection function determines whether or not there is an abnormality in the flight state of the unmanned aircraft 11 based on telemetry information (abnormality detection step S41).

[0076] The important position determination function includes position information regarding the position of the unmanned aircraft 11 in the formation of the formation, and determines whether the unmanned aircraft 11 is positioned in an important position in the formation based on the individual identification information (important position determination step S46).

[0077] The transmission function sends a command to the radio aircraft in formation flight to replace the unmanned aircraft 11 that has been determined to be positioned in an important location and is deemed to have an abnormality with an unmanned aircraft 11 that has been determined not to be positioned in an important location and is deemed to have no abnormality (transmission step S47).

[0078] (Regarding methods for monitoring unmanned aerial vehicles and other embodiments of unmanned aerial vehicle monitoring programs) Next, with reference to Figure 7, an unmanned aerial vehicle (UAV) monitoring program according to another embodiment of the present disclosure will be described along with an UAV monitoring method according to another embodiment. Figure 7 is an example of a flowchart of an UAV monitoring program according to another embodiment. The flowchart of the unmanned aerial vehicle monitoring program according to another embodiment shown in Figure 7 differs from the flowchart of the unmanned aerial vehicle monitoring program shown in Figure 6 in that it includes the addition of the captured image acquisition step S42, the learning model acquisition step S43, the depiction anomaly determination step S44, and the anomaly extraction step S45. The unmanned aerial vehicle monitoring method according to other embodiments is executed by the processing unit 10e of the unmanned aerial vehicle monitoring system 10 based on the unmanned aerial vehicle monitoring program according to other embodiments shown in Figure 7. The unmanned aerial vehicle monitoring program according to another embodiment shown in Figure 7 includes a telemetry information acquisition step S40, an anomaly detection step S41, a captured image acquisition step S42, a learning model acquisition step S43, a depiction anomaly detection step S44, an anomaly extraction step S45, an important position determination step S46, and a transmission step S47, among others.

[0079] The unmanned aerial vehicle monitoring program according to another embodiment shown in Figure 7 enables the processing unit 10e of the unmanned aerial vehicle monitoring system 10 to perform functions such as telemetry information acquisition, anomaly detection, image acquisition, learning model acquisition, depiction anomaly detection, anomaly extraction, important position determination, and transmission. These functions are executed in the order shown in the flowchart of Figure 7, but the order can be changed as appropriate. Below, we will describe the unmanned aerial vehicle monitoring method and unmanned aerial vehicle monitoring program according to another embodiment shown in Figure 7, focusing only on the differences from the unmanned aerial vehicle monitoring method and unmanned aerial vehicle monitoring program shown in Figure 6. Furthermore, since each function overlaps with the descriptions of the various functional units of the unmanned aerial vehicle monitoring system 10 mentioned above, detailed explanations will be omitted.

[0080] The image acquisition function acquires images captured by a camera that captures the scene drawn by multiple unmanned aircraft 11 during formation flight (image acquisition step S42).

[0081] The learning model acquisition function acquires a learning model that has been pre-trained on normal depictions, determines whether or not there are any abnormalities in the depictions contained in the captured image, and, if it is determined that there are abnormalities in the depictions, outputs the individual identification information of the unmanned aircraft 2 that caused the abnormality (learning model acquisition step S43).

[0082] The image abnormality detection function determines whether or not there is an image abnormality based on the captured image (image abnormality detection step S44).

[0083] The anomaly detection function extracts individual identification information of the unmanned aircraft 2 that caused the anomaly when it is determined that the depiction is an anomaly (anomaly detection step S45).

[0084] (Effects of the unmanned aerial vehicle monitoring system 10) The unmanned aerial vehicle monitoring system 10 according to this embodiment can quickly identify the cause of an anomaly and automatically take necessary measures when an anomaly occurs in multiple unmanned aerial vehicles during formation flight. Specifically, types of anomalies include position anomalies, altitude anomalies, battery level anomalies, attitude anomalies, approaching obstacles, and communication anomalies. When these anomalies are detected, the following measures will be taken. Position repositioning: If a positional or altitude anomaly is detected, the formation will maintain its position by repositioning other normal unmanned aircraft to the location of the anomaly. Emergency Return Command: In the event of a battery level abnormality, a return command will be issued to the abnormal unmanned aircraft, ensuring safety by having it withdraw from formation flight. Attitude correction: When an attitude anomaly occurs, the control unit of the unmanned aerial vehicle attempts to correct its attitude, and if correction is difficult, it sends a command to change its position in a way that does not affect surrounding unmanned aerial vehicles. Obstacle Avoidance Command: If an obstacle is detected approaching, the unmanned aircraft will be instructed to take a new flight path to avoid the obstacle, thus maintaining the safety of the formation. Communication recovery and information supplementation: In the event of a communication anomaly, attempts will be made to restore communication while simultaneously inferring the location and status of the anomaly based on indirect information from other unmanned aerial vehicles, and monitoring will continue. These measures minimize the impact of a malfunctioning unmanned aerial vehicle (UAV) on the formation and its performance. Furthermore, by compensating for the malfunctioning vehicle's position and role with a functioning UAV, the overall stability and safety of the formation can be ensured, mitigating disruptions to the performance and negative impacts on flight performance caused by multiple UAVs.

[0085] Specifically, the telemetry information acquisition unit 40 acquires flight status information such as altitude, attitude, battery level, and position coordinates of each unmanned aircraft in real time, enabling early detection of anomalies. For example, if the altitude deviates from the set range, if the battery level drops to a dangerous level, or if the attitude is unstable and abnormal tilting or shaking is detected, all of these are detected immediately. Furthermore, because this data is acquired continuously, even in cases where an anomaly progresses gradually, it is possible to grasp the initial signs and predict the occurrence of an anomaly early on. The abnormality detection unit 41 automatically determines whether each unmanned aircraft has an abnormality based on the acquired telemetry information, enabling rapid detection of abnormalities. Furthermore, the critical position detection unit 46 determines whether the unmanned aircraft 2 that has experienced an abnormality is located within the formation, thereby minimizing the impact of an abnormality in one unmanned aircraft on the flight path of multiple unmanned aircraft and ensuring the stability and safety of the formation.

[0086] Furthermore, if the malfunctioning unmanned aerial vehicle 2 is located in a critical position, the transmitter 47 commands other normal unmanned aerial vehicles to take its place, thereby mitigating the impact of the malfunction in unmanned aerial vehicle 11 on the depiction of multiple unmanned aerial vehicles. Through this series of functions, the unmanned aerial vehicle monitoring system 10 achieves consistent monitoring and control from malfunction detection to response commands, minimizing the impact of malfunctions and reducing the impact on depiction even if a malfunction occurs in an unmanned aerial vehicle during formation flight.

[0087] Furthermore, by combining the image acquisition unit 42 and the depiction anomaly detection unit 44, it becomes possible to visually confirm anomalies in the formation shape and positional relationships, and to accurately detect anomalies that could not be detected with conventional telemetry information alone. As a result, the anomaly extraction unit 45 identifies the individual identification information of the unmanned aircraft 2 that is causing the anomaly, and the cause can be eliminated quickly, further improving the safety and efficiency of formation flight.

[0088] Thus, the unmanned aerial vehicle monitoring system 10 of this embodiment has the effect of significantly improving the stability and safety of formation flight by automatically detecting abnormalities in the unmanned aerial vehicles 11 during formation flight and enabling immediate response.

[0089] Furthermore, if, for example, a camera installed on the ground captures an image of a formation of multiple unmanned aircraft (UAVs) in flight, and the UAV 11 causing the anomaly is in a critical position, the UAV 11 showing the anomaly can be replaced with an UAV 11 that is not in a critical position and is determined to be free of anomalies. Thus, the UAV monitoring system 10 can detect anomalies in UAVs 11 that cannot be detected by telemetry information, and can mitigate the impact on the image even if an anomaly occurs in an UAV during formation flight.

[0090] This disclosure is not limited to the unmanned aerial vehicle monitoring system 10, unmanned aerial vehicle monitoring method, and unmanned aerial vehicle monitoring program according to the embodiments described above. It can be implemented in various other modifications or applications without departing from the gist of this disclosure as described in the claims. Furthermore, although the above embodiment uses the term "data," the term "data" can be replaced with "information," and the term "information" can be replaced with "data." [Explanation of Symbols]

[0091] 1. Description 2. Unmanned aircraft exhibiting abnormal behavior 3. Normal unmanned aircraft 4a GPS signal 4b GPS satellite 5 Wireless communication antenna 6 RTK reference station 7. Position where the unmanned aerial vehicle is missing. 10 Unmanned Aerial Vehicle Monitoring System 10a Communications Department 10b ROM 10c RAM 10d storage section 10e Processing Unit 10f Input / Output Interface 10g display 10h Operation Input Section 10i External Information Input Section 11 Unmanned aircraft 17. Information and Communication Networks 18 Main body 19 Arms 19a First Arm 19b Second Arm 19th Third Arm 19d Fourth Arm 20 propellers 20a No. 1 propeller 20b Second propeller 20c Third propeller 20d Fourth propeller 21 Light-emitting part 22 Communications Department 23 ROM 24 RAM 25 Memory section 26 Control Unit 26a Acquisition Department 26b Anomaly detection unit 26c Abnormality notification section 26d Light Insufficiency Detection Unit 26e Insufficient light notification section 26f Luminous intensity correction section 27 Batteries 28 Input / Output Interfaces 29. Flight drive unit (motor) 30. Flight propulsion mechanism (rotor) 31 GPS receivers 32 Illuminance Sensor 33 Geomagnetic Sensor 34. Altitude Sensor 35 Gyroscope Sensor 36 Obstacle detection cameras 40 Telemetry Information Acquisition Unit 41 Abnormality determination section 42 Image acquisition unit 43 Learning Model Acquisition Unit 44 Determination of abnormality in depiction 45 Abnormality extraction part 46. ​​Important Position Determination Unit 47 Transmitter S40 Telemetry Information Acquisition Step S41 Anomaly detection step S42 Image acquisition step S43 Steps to acquire a learning model S44 Step for determining whether there is a depiction anomaly S45 Anomaly detection step S46 Important Position Determination Step S47 Transmission Step

Claims

1. An unmanned aerial vehicle monitoring system that monitors the flight status of multiple unmanned aerial vehicles in formation flight, A telemetry information acquisition unit acquires telemetry information indicating the flight status of the unmanned aircraft during formation flight, which is transmitted from each of the unmanned aircraft and associated with individual identification information that is pre-assigned to the plurality of unmanned aircraft and identifies each individual unmanned aircraft. An abnormality determination unit that determines whether or not there is an abnormality in the flight state of the unmanned aircraft based on the telemetry information, The individual identification information includes position information relating to the position of the unmanned aircraft in forming a formation, and a critical position determination unit determines whether or not the unmanned aircraft is positioned in an important position in forming a formation based on the individual identification information. A transmitting unit that transmits a command to a radio-controlled aircraft in formation flight to replace the unmanned aircraft that has been determined to be positioned in the aforementioned important position and has been determined to have the aforementioned abnormality with the unmanned aircraft that has been determined not to be positioned in the aforementioned important position and has been determined to have the aforementioned abnormality, An unmanned aerial vehicle monitoring system characterized by having the following features.

2. The image acquisition unit acquires images captured by a camera that captures the depiction drawn by the multiple unmanned aircraft during the formation flight, A depiction abnormality determination unit that determines whether or not there is an abnormality in the depiction based on the captured image, If the aforementioned description is determined to be abnormal, an abnormality extraction unit extracts individual identification information of the unmanned aircraft that caused the abnormality, Furthermore, The critical position determination unit determines whether the unmanned aircraft related to the individual identification information is positioned in a critical position based on the individual identification information extracted by the abnormality extraction unit. The aforementioned transmitting unit The unmanned aircraft monitoring system according to claim 1, characterized in that, when the abnormality detection unit determines that the unmanned aircraft related to the individual identification information it has extracted is positioned in an important position, it transmits a command to the radio-controlled aircraft in formation flight to replace the unmanned aircraft with an unmanned aircraft that has been determined not to be positioned in an important position and has been determined not to have any abnormalities.

3. The unmanned aircraft monitoring system according to claim 1, characterized in that the telemetry information includes at least one of the altitude, temperature, battery level, attitude, and position coordinates of the unmanned aircraft.

4. The system further includes a learning model acquisition unit that acquires a learning model that pre-learns normal depictions, determines whether or not there are abnormalities in the depictions included in the captured image, and, if it is determined that there are abnormalities in the depictions, outputs individual identification information of the unmanned aerial vehicle that is the cause of the abnormality. The depiction abnormality determination unit uses the learning model to determine whether or not there is a depiction abnormality in the captured image, The unmanned aircraft monitoring system 10 according to claim 2, characterized in that the anomaly extraction unit extracts individual identification information of the unmanned aircraft that is the cause of the anomaly contained in the captured image using the learning model.

5. The unmanned aircraft monitoring system according to claim 4, characterized in that the learning model pre-learns the shape and brightness of the contour of a normal depiction, determines whether or not there is an abnormality in the depiction included in the captured image, and outputs individual identification information of the unmanned aircraft that caused the abnormality when it is determined that there is an abnormality in the depiction included in the captured image.

6. An unmanned aerial vehicle monitoring method used in an unmanned aerial vehicle monitoring system 10 that monitors the flight status of multiple unmanned aerial vehicles in formation flight, The aforementioned unmanned aircraft monitoring system, A telemetry information acquisition step involves acquiring telemetry information indicating the flight status of the unmanned aircraft during formation flight, which is transmitted from each of the multiple unmanned aircraft and associated with individual identification information that is pre-assigned to each of the unmanned aircraft and identifies each individual unmanned aircraft; An abnormality determination step in which an abnormality in the flight state of the unmanned aircraft is determined based on the telemetry information, The individual identification information includes position information relating to the position of the unmanned aircraft in forming a formation, and the important position determination step determines whether the unmanned aircraft is positioned in an important position in forming a formation based on the individual identification information. A transmission step of transmitting a command to a radio aircraft in formation flight to replace the unmanned aircraft that has been determined to be positioned in the aforementioned important position and has been determined to have the aforementioned abnormality with the unmanned aircraft that has been determined not to be positioned in the aforementioned important position and has been determined to have the aforementioned abnormality, A method for monitoring unmanned aircraft, characterized by causing the system to perform the following actions.

7. An unmanned aerial vehicle (UAV) monitoring program used in an UAV monitoring system that monitors the flight status of multiple UAVs in formation flight, The aforementioned unmanned aircraft monitoring system, A telemetry information acquisition function that acquires telemetry information indicating the flight status of the unmanned aircraft during formation flight, which is transmitted from each of the multiple unmanned aircraft and associated with individual identification information that is pre-assigned to each of the unmanned aircraft and identifies each individual unmanned aircraft, An abnormality detection function that determines whether or not there is an abnormality in the flight state of the unmanned aircraft based on the telemetry information, The individual identification information includes position information relating to the position of the unmanned aircraft in forming a formation, and a critical position determination function determines whether the unmanned aircraft is positioned in an important position in forming a formation based on the individual identification information. A transmission function that transmits a command to a radio-controlled aircraft in formation flight to replace the unmanned aircraft that has been determined to be positioned in the aforementioned important position and has been determined to have the aforementioned abnormality with the unmanned aircraft that has been determined not to be positioned in the aforementioned important position and has been determined to have the aforementioned abnormality, An unmanned aerial vehicle monitoring program characterized by achieving this.

Images