Imaging system using unmanned aircraft, imaging method, and program

Abstract

Provided is a technique and the like with which it is possible to ensure safety when tracking and imaging competitors in relation to a technique for aerial photography using an unmanned aircraft. This imaging system comprises a plurality of unmanned aircraft 1 constituting a formation 10 that fly while tracking a player P who is an imaging target. Each unmanned aircraft 1 images the imaging target by means of a camera and provides a captured image. The plurality of unmanned aircraft 1 fly through a formation flight position in a prescribed positional relationship with respect to the imaging target. At least one unmanned aircraft among the plurality of unmanned aircraft 1 controls flight of the unmanned aircraft so that a parameter value for at least one of a relative position, a relative distance, an absolute position, a relative speed, and an absolute speed with respect to the imaging target satisfies a prescribed condition. Any unmanned aircraft 1 that does not satisfy the prescribed condition is determined to be a formation flight abnormality, and the unmanned aircraft 1 that is the formation flight abnormality is moved in a direction away from the imaging target.

Description

Photography system, photography method, and program using unmanned aerial vehicle

[0001] The present disclosure relates to technology using unmanned aerial vehicles (UAVs) such as drones.

[0002] Technology is being developed to photograph athletes and others using cameras mounted on unmanned aerial vehicles such as drones.

[0003] An example of the prior art is Japanese Patent No. 6885485 (Patent Document 1), which describes a system and method for capturing still and / or moving scenes using a multiple camera network.

[0004] Patent No. 6885485

[0005] As in the example of Patent Document 1, there is a technology for controlling the position and direction of multiple unmanned aerial vehicles flying in formation to surround a subject to be photographed, such as an athlete.

[0006] In competitions, races, sports, etc., in which competitors run on a predetermined course, there is a demand for a method of tracking and filming competitors so that viewers and commentators can easily grasp the movements, expressions, rankings, etc. of the competitors / players.

[0007] Conventionally, in the case of, for example, a motor race, a method of taking pictures using a camera installed at a fixed position on the course has been used, while in the case of, for example, a marathon, a method of taking pictures using a camera running alongside the runners has been used. In the former case, the fixed camera has limited timing and range of shooting, requires multiple cameras, and makes it difficult to constantly follow the runners. In the latter case, in the case of a side-by-side camera, it is difficult to follow the runners in events held on unpaved courses (for example, snowboarding) or events where the runners move at high speeds (for example, motor racing), because it is difficult to have the cameras running alongside the runners.

[0008] In recent years, unmanned aerial vehicles such as drones have become more popular, and it is now possible to film athletes from the air using either manned or unmanned aircraft. This aerial photography method allows for the capture of a wide area of ​​the course, eliminating the need for multiple cameras. With this method, even in races held on unpaved courses or where the athletes are moving at high speeds, it is possible to continuously capture the athletes' movements and expressions by tracking them through the air.

[0009] When capturing and distributing video footage, various factors are required, such as the preferred angle of view and distance, depending on the sport. For example, footage capturing the athlete from the front, from the left and right sides, or from behind may be required. During a competition, athletes are constantly moving, and their position, speed, and direction of movement fluctuate. It is difficult to capture footage with such a preferred angle of view using conventional methods using fixed cameras or parallel cameras. It is particularly difficult to simultaneously capture footage with multiple different angles of view.

[0010] Therefore, it is desirable to apply an aerial photography method for photographing athletes using unmanned aerial vehicles and control the formation flight of multiple unmanned aerial vehicles so that they follow the athletes, thereby enabling the acquisition of footage with a desirable shooting angle of view, etc. In particular, it is desirable to be able to follow the athletes using multiple unmanned aerial vehicles flying in formation and simultaneously acquire footage with multiple different shooting angles of view, etc.

[0011] The above-mentioned method of aerial photography using unmanned aerial vehicles presents the following challenges. First and foremost, safety must be ensured. It is necessary to reduce the risk of contact or collision between unmanned aerial vehicles and competitors, spectators, and other people. To ensure this safety, formation flight control is required. For example, even if an unmanned aerial vehicle becomes abnormal, it is necessary to be able to more reliably avoid contact or collision with competitors.

[0012] Furthermore, it is desirable to ensure safety and to improve the quality of the footage captured by the unmanned aerial vehicle as much as possible. For example, it is desirable to be able to obtain footage with a favorable angle of view, etc., as described above, so that the distance of the unmanned aerial vehicle from the competitor can be stabilized and the competitor's movements can be clearly understood.

[0013] The purpose of the present disclosure is to provide aerial photography technology using the above-mentioned unmanned aerial vehicle that can ensure safety and improve the quality of the captured footage when following and photographing athletes.

[0014] Representative embodiments of the present disclosure are an imaging system, an imaging method, and the like, and are characterized by having the following configurations.

[0015] (1) A photography system using unmanned aerial vehicles, comprising a plurality of unmanned aerial vehicles that fly in pursuit of a subject to be photographed, each of which photographs the subject with a camera to provide photographed footage, the plurality of unmanned aerial vehicles flying in a formation flight position with a predetermined positional relationship to the subject to be photographed, and at least one first unmanned aerial vehicle among the plurality of unmanned aerial vehicles controlling the flight of the first unmanned aerial vehicle so that at least one parameter value of the relative position, relative distance, absolute position, relative velocity, and absolute velocity with respect to the subject to be photographed satisfies a predetermined condition, and determining that a unmanned aerial vehicle that does not satisfy the predetermined condition is engaging in formation flight abnormality, and taking evasive action to move the unmanned aerial vehicle that is engaging in formation flight abnormality in a direction away from the subject to be photographed.

[0016] (2) In the imaging system of (1), the evacuation behavior includes at least movement in a direction increasing the altitude.

[0017] (3) In the photographing system of (1) or (2), the first unmanned aerial vehicle acquires position information of the subject to be photographed and controls the position of the first unmanned aerial vehicle so that the relative position or relative distance of the first unmanned aerial vehicle to the position of the subject to be photographed satisfies a target value or target range.

[0018] (4) An imaging system according to any one of (1) to (3), wherein all of the plurality of unmanned aerial vehicles are the first unmanned aerial vehicle.

[0019] (5) In any of the photographing systems (1) to (4), the first unmanned aerial vehicle acquires information on the position and movement direction of the subject to be photographed, and controls the position of the first unmanned aerial vehicle so that the relative position or relative distance of the first unmanned aerial vehicle to the position and movement direction of the subject to be photographed satisfies a target value or target range.

[0020] (6) In any of the photographing systems (1) to (5), when the relative position or relative distance of the first unmanned aerial vehicle to the subject to be photographed does not satisfy the target value or target range, or when the difference between the absolute position of the first unmanned aerial vehicle and the target absolute position exceeds a predetermined value, the photographing system determines that the first unmanned aerial vehicle is in formation flight abnormality.

[0021] (7) In any of the photographing systems (1) to (6), when the relative speed of the first unmanned aerial vehicle with respect to the subject to be photographed does not satisfy the target value or target range, or when the difference between the absolute speed of the first unmanned aerial vehicle and the target absolute speed exceeds a predetermined value, the photographing system determines that the first unmanned aerial vehicle is in formation flight abnormality.

[0022] (8) In any of the photography systems (1) to (7), a second unmanned aerial vehicle among the plurality of unmanned aerial vehicles, which is not the first unmanned aerial vehicle, acquires position information of the first unmanned aerial vehicle and controls the position of the second unmanned aerial vehicle so that the relative position or relative distance of the second unmanned aerial vehicle to the position of the first unmanned aerial vehicle satisfies a target value or target range.

[0023] (9) In any of the photography systems (1) to (8), a second unmanned aerial vehicle among the plurality of unmanned aerial vehicles, which is not the first unmanned aerial vehicle, acquires position information and information on the direction of movement of the first unmanned aerial vehicle, and controls the position of the second unmanned aerial vehicle so that the relative position or relative distance of the second unmanned aerial vehicle to the position and direction of movement of the first unmanned aerial vehicle satisfies a target value or target range.

[0024] (10) An imaging system according to any one of (1) to (9), which determines that the second unmanned aerial vehicle is experiencing a formation flight abnormality when the relative position or relative distance of the second unmanned aerial vehicle to the first unmanned aerial vehicle does not satisfy the target value or target range.

[0025] (11) An imaging system according to any one of (1) to (10), which determines that the second unmanned aerial vehicle is in formation flight abnormality when the relative speed of the second unmanned aerial vehicle with respect to the first unmanned aerial vehicle does not satisfy a target value or target range, or when the difference between the absolute speed of the second unmanned aerial vehicle and the target absolute speed exceeds a predetermined value.

[0026] (12) An imaging system in any one of (1) to (11), in which the unmanned aerial vehicle in which a control communication abnormality or a positioning accuracy abnormality is detected is judged to be in formation flight abnormality, or the first unmanned aerial vehicle in which a positioning accuracy abnormality is detected of the subject to be imaged is judged to be in formation flight abnormality.

[0027] (13) A photography system according to any one of (1) to (12), in which, when the first unmanned aerial vehicle is determined to be flying abnormally in the formation, the entire formation consisting of the plurality of unmanned aerial vehicles, including the first unmanned aerial vehicle, performs the evacuation action.

[0028] (14) In any of the photography systems (1) to (13), when the second unmanned aerial vehicle is determined to be in the formation flight abnormality, the photography system causes the second unmanned aerial vehicle that is in the formation flight abnormality to perform the evacuation action, or causes the entire formation consisting of the multiple unmanned aerial vehicles including the second unmanned aerial vehicle that is in the formation flight abnormality to perform the evacuation action.

[0029] (15) In any one of the photography systems (1) to (14), the orientation of the camera of the unmanned aerial vehicle is controlled so that the subject to be photographed appears near the center of the image captured by the camera.

[0030] (16) In any of the photographing systems (1) to (15), the position of the subject to be photographed is determined from a sensor provided on the subject, or the position of the subject to be photographed is determined by a sensor provided on the unmanned aerial vehicle, or the position of the subject to be photographed is determined by a sensor provided in the environment in which the subject to be photographed moves, or the position of the subject to be photographed is determined from the image captured by the camera.

[0031] (17) In any of the photographing systems (1) to (16), the direction of movement of the subject to be photographed is determined from a sensor provided on the subject, or the direction of movement of the subject to be photographed is determined from a sensor provided on the unmanned aerial vehicle, or the direction of movement of the subject to be photographed is determined from a sensor provided in the environment in which the subject to be photographed moves, or the direction of movement of the subject to be photographed is determined from the image captured by the camera.

[0032] (18) In any of the photography systems (1) to (17), the formation flight positions of the multiple unmanned aerial vehicles include at least two of the following: a forward position in a forward direction relative to the subject to be photographed, a rear position in a rearward direction relative to the subject to be photographed, a right position in a rightward direction relative to the subject to be photographed, a left position in a leftward direction relative to the subject to be photographed, and a directly above position in a directly above direction relative to the subject to be photographed, photography system.

[0033] (19) In any of the photography systems (1) to (18), the escape behavior includes the following movements depending on the formation flight position: in the case of the unmanned aerial vehicle in the front position, movement in the direction of increasing altitude, movement to the left or right, or movement backward; in the case of the unmanned aerial vehicle in the rear position, movement in the direction of increasing altitude, movement to the left or right, or movement backward; in the case of the unmanned aerial vehicle in the right position, movement in the direction of increasing altitude, movement to the right, or movement backward; in the case of the unmanned aerial vehicle in the left position, movement in the direction of increasing altitude, movement to the left, or movement backward; in the case of the unmanned aerial vehicle in the directly overhead position, movement in the direction of increasing altitude, movement to the left or movement backward.

[0034] (20) A method for photographing in a photography system using unmanned aerial vehicles, the photography system comprising a plurality of unmanned aerial vehicles that fly in pursuit of a subject to be photographed, each of the plurality of unmanned aerial vehicles photographing the subject to be photographed with a camera to provide photographed footage, the plurality of unmanned aerial vehicles flying in a formation flight position in a predetermined positional relationship with the subject to be photographed, the photography method comprising the steps of: controlling the flight of at least one unmanned aerial vehicle among the plurality of unmanned aerial vehicles so that at least one parameter value of the relative position, relative distance, absolute position, relative velocity, and absolute velocity with respect to the subject to be photographed satisfies a predetermined condition; determining that an unmanned aerial vehicle that does not satisfy the predetermined condition is in abnormal formation flight; and moving the unmanned aerial vehicle in the abnormal formation flight direction away from the subject to be photographed.

[0035] (21) A program for causing a computer to execute processing corresponding to the photographing system of (1) or the photographing method of (20), or a storage medium / recording medium storing the program.

[0036] According to a representative embodiment of the present disclosure, it is possible to provide a technology for aerial photography using the unmanned aerial vehicle that can ensure safety and improve the quality of the captured video when tracking and filming athletes. Problems, configurations, effects, etc. other than those described above are described in the description of the embodiment of the invention.

[0037] 1 is a diagram showing the overall system configuration in the first embodiment; 2 is a diagram showing the configuration of the formation flight position of the formation in the first embodiment; 3 is a plan view showing an example of the formation flight position of the formation, the leader aircraft, and the sub aircraft as viewed from above in the first embodiment; 4 is a plan view showing the formation flight position of the formation as viewed from the side in the first embodiment; 5 is a plan view showing the formation flight position of the formation as viewed from the top in the first embodiment; 6 is a plan view showing the formation tracking control as viewed from the top in the first embodiment; 7 is a plan view showing another example of the formation flight position of the formation as viewed from the side in the first embodiment; 8 is a plan view showing another example of the formation flight position of the formation as viewed from the top in the first embodiment; 9 is an explanatory diagram of the first formation tracking control in the first embodiment; 10 is an explanatory diagram of the second formation tracking control in the first embodiment; 11 is a plan view showing an example of the setting of the leader aircraft of the formation as viewed from above in the first embodiment; 12 is a plan view showing additional functions as viewed from above in the first embodiment; 13 is a functional block configuration diagram of an unmanned aerial vehicle (drone) in the first embodiment; 14 is a diagram showing an example of the appearance of an unmanned aerial vehicle (drone) in the first embodiment; 1 is a functional block diagram of a control device in embodiment 1. FIG. 2 is a diagram showing an example of the external appearance of the control device in embodiment 1. FIG. 3 is a functional block diagram of a pre-setting terminal in embodiment 1. FIG. 4 is a functional block diagram of a captured image display device in embodiment 1. FIG. 5 is a functional block diagram of a competitor terminal in embodiment 1. FIG. 6 is the first half of a functional block diagram of a server in embodiment 1. FIG. 7 is the second half of a functional block diagram of a server in embodiment 1. FIG. 8 is a state transition diagram in embodiment 1. FIG. 9 is a flow diagram of a pre-takeoff preparation mode in embodiment 1. FIG. 10 is a diagram showing a first example of a standby position in embodiment 1. FIG. 11 is a diagram showing a second example of a standby position in embodiment 1. FIG. 12 is a diagram showing a third example of a standby position in embodiment 1. FIG. 13 is a diagram showing an example of a geofence in embodiment 1. FIG. 14 is a diagram showing an example screen display of a captured image display device in embodiment 1. FIG. 15 is a flow diagram of a capture standby mode in embodiment 1. FIG. 16 is a flow diagram of a follow-up capture mode in embodiment 1. FIG. 17 is a flow diagram of formation flight abnormality determination for a leader aircraft in embodiment 1. FIG. 18 is a flow diagram of formation flight abnormality determination for a sub aircraft in embodiment 1. FIG. 1 is an explanatory diagram of technical means for grasping the position of an athlete in the first embodiment. FIG. 2 is an explanatory diagram of communication abnormalities in the first embodiment. FIG. 3 is an explanatory diagram of determining the relative distance to an athlete in the first embodiment. FIG. 4 is an explanatory diagram of determining proximity to a group of athletes in the first embodiment. FIG. 5 is an explanatory diagram of remote determination in the first embodiment. FIG. 6 is a flow diagram of the evacuation mode in the event of an emergency in the first embodiment. FIG. 7 is an explanatory diagram of a first example of evacuation in the evacuation mode in the event of an emergency in the first embodiment.1 is an explanatory diagram of a second example of evacuation in the abnormality evacuation mode in the first embodiment. FIG. 2 is an explanatory diagram of an evacuation action in which following is stopped in the first embodiment. FIG. 3 is an explanatory diagram of an evacuation action in which following is maintained in the first embodiment. FIG. 4 is a diagram showing an example of an evacuation action in a forward position in the first embodiment. FIG. 5 is a diagram showing an example of an evacuation action in a left or right position in the first embodiment. FIG. 6 is a diagram showing an example of an evacuation action in a rear position in the first embodiment. FIG. 7 is a diagram showing an example of an evacuation action in a directly overhead position in the first embodiment. FIG. 8 is a flow diagram of the landing mode in the first embodiment. FIG. 9 is an explanatory diagram of determining an evacuation target in response to an abnormality in the first embodiment. FIG. 10 is an explanatory diagram of determining an evacuation target as a modified example in the first embodiment. FIG. 11 is an explanatory diagram showing an example of priority related to formation flight positions in the first embodiment. FIG. 12 is an explanatory diagram showing an example of control using priority in the first embodiment. FIG. 13 is an explanatory diagram showing temporary evacuation and return as a modified example in the first embodiment. FIG. 14 is an explanatory diagram showing a combination of evacuation and camera zoom control as a modified example in the first embodiment.

[0038] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the drawings, the same parts are generally designated by the same reference numerals, and repeated explanations will be omitted. In the drawings, the representation of components may not represent their actual positions, sizes, shapes, ranges, etc., in order to facilitate understanding of the invention, but this is not intended to limit the scope of the invention.

[0039] For the purpose of explanation, when describing processing by a program, the program, functions, processing units, etc. may be described as the main components, but the main hardware components are the processor, or a controller, device, computer, system, etc. that is configured with the processor, etc. The computer executes processing according to the program read into memory using resources such as memory and communication interfaces as appropriate through the processor. This realizes predetermined functions, processing units, etc. The processor is configured, for example, with semiconductor devices such as a CPU / MPU or GPU. Processing is not limited to software program processing, and can also be implemented using dedicated circuits. Dedicated circuits such as FPGAs, ASICs, and CPLDs can be used.

[0040] The program may be pre-installed as data on the target computer, or may be distributed as data from a program source to the target computer. The program source may be a program distribution server on a communication network, or a non-transitory computer-readable storage medium, such as a memory card or disk. The program may be composed of multiple modules. The computer system may be composed of multiple devices. The computer system may be composed of a client-server system, a cloud computing system, an IoT system, etc. Various data and information may be composed of structures such as tables and lists, for example, but are not limited to these. Expressions such as identification information, identifiers, IDs, names, and numbers are interchangeable.

[0041] [Issues and Solutions] To further discuss the issues and solutions, as mentioned above, the filming method using unmanned aerial vehicles requires both 1. ensuring safety and 2. high quality of the filmed images.

[0042] 1. From the perspective of ensuring safety, even when multiple unmanned aerial vehicles are used to follow athletes during a competition and photograph them from a relatively close distance, it is necessary to: 1A. reduce the risk of collision with the athletes being photographed, and 1B. reduce the risk of collision between unmanned aerial vehicles.

[0043] 2. From the perspective of quality of the footage, even when multiple unmanned aerial vehicles are used to follow athletes during a competition and film them from a relatively close distance, it is necessary to continue filming athletes whose movements are fast and whose position and direction of movement change so that they appear at an optimal angle of view, position, and size. To achieve this, it is necessary to 2A. accurately grasp and follow the athletes' movements, and 2B. control the camera according to the athletes' positions, etc.

[0044] 1A. Reducing the risk of collision with the competitor being filmed: 1A-1. The filming system uses sensors, etc. to grasp the status regarding the risk of contact with the competitor. The filming system determines that there is a formation flight abnormality if the relative position or relative distance of the unmanned aerial vehicle to the competitor falls outside a predetermined target range, or if the difference between the unmanned aerial vehicle's current absolute position and the target absolute position exceeds a predetermined value. The target range ranges from a predetermined lower limit value to a predetermined upper limit value. Outside the target range means being below the lower limit value or exceeding the upper limit value. A formation flight abnormality is a state in which the flight control of the formation is abnormal. In other words, an abnormality is when the unmanned aerial vehicle is too close or too far from the competitor.

[0045] 1A-2. The filming system also determines that there is an abnormality in the formation flight if the relative speed of the unmanned aircraft with respect to the competitor falls outside a predetermined target range for speed, or if the difference between the current absolute speed and the target speed exceeds a predetermined value. In other words, it determines that there is an abnormality when the response speed or movement speed of the unmanned aircraft is insufficient.

[0046] 1A-3. In addition, if the photography system detects a control communication abnormality or a positioning accuracy abnormality of the unmanned aircraft between the competitor and the unmanned aircraft, it determines that the unmanned aircraft is experiencing a formation flight abnormality. In addition, if the photography system detects a positioning accuracy abnormality of the competitor in the unmanned aircraft, it determines that the unmanned aircraft is experiencing a formation flight abnormality.

[0047] 1B. Reducing the risk of collision between unmanned aerial vehicles: 1B-1. The photography system uses sensors and other devices to understand the status of the risk of contact between unmanned aerial vehicles in a formation and other unmanned aerial vehicles. If the relative position or relative distance between a certain unmanned aerial vehicle and other unmanned aerial vehicles falls outside the target range, the photography system determines that there is an abnormality in the formation flight of that unmanned aerial vehicle. In other words, if a certain unmanned aerial vehicle is too close or too far away from other unmanned aerial vehicles, it determines that there is an abnormality.

[0048] 1B-2. The imaging system also determines that there is a formation flight abnormality when the relative speed of the unmanned aerial vehicle relative to other unmanned aerial vehicles falls outside a predetermined target range of speed, or when the difference between the current absolute speed and the target speed exceeds a predetermined value.

[0049] 1B-3. Furthermore, when the imaging system detects a control communication abnormality or a positioning accuracy abnormality between unmanned aerial vehicles, it determines that the unmanned aerial vehicles are experiencing a formation flight abnormality.

[0050] 2A. Accurately Understanding and Tracking Athlete Movements: 2A-1. Understanding Athlete ID and Location: The filming system wirelessly acquires information from the athlete's terminal / sensor equipped on the athlete. The athlete's terminal / sensor information may include GNSS positioning information as the athlete's ID and location information. Alternatively, the filming system may acquire the athlete's ID and location information using a sensor (such as a ranging sensor) equipped on the unmanned aerial vehicle. Alternatively, the filming system may acquire the athlete's ID and location information from a device equipped in the competition environment. Alternatively, the filming system may estimate the athlete's ID and location from the footage captured by the unmanned aerial vehicle's camera and camera control information (such as the camera's position and orientation).

[0051] 2A-2. Understanding the Athlete's Movement Direction and Speed: The filming system may acquire information on the athlete's movement direction and speed from the athlete's terminal / sensor equipped on the athlete. Alternatively, the filming system may acquire information on the athlete's movement direction and speed using a sensor (such as a distance sensor) equipped on the unmanned aerial vehicle. Alternatively, the filming system may acquire information on the athlete's movement direction and speed from a device equipped in the competition environment. Alternatively, the filming system may estimate the athlete's movement direction and speed from the video footage captured by the unmanned aerial vehicle's camera and camera control information.

[0052] 2B. Controlling the camera according to the competitor's position, etc.: 2B-1. Stabilizing the sense of distance to the competitor in the captured image: The filming system controls the position of the unmanned aerial vehicle so that the relative distance between the competitor and the unmanned aerial vehicle remains constant, in other words, so that the distance remains within a target range or is close to a target value. The filming system controls camera parameter values ​​such as the camera direction and zoom amount according to the relative distance between the competitor and the unmanned aerial vehicle so that the sense of distance and size of the competitor in the captured image remains constant, or so that the sense of distance and size are appropriate.

[0053] 2B-2. Stabilizing the angle of view of the competitor in the captured image: The imaging system controls the position of the unmanned aircraft (i.e., the formation flight position) so that the relative position of the unmanned aircraft to the position of the competitor is constant. The imaging system also controls the orientation of the camera of the unmanned aircraft according to the relative position of the unmanned aircraft.

[0054] 2B-3. Stabilizing the position of the athlete in the captured image: The filming system controls the orientation of the camera of the unmanned aerial vehicle so that the athlete is captured in an optimal position, such as near the center, in the captured image, depending on the relative position of the unmanned aerial vehicle.

[0055] 3. Regarding the generation of control targets for formation flight: The filming system generates control target values ​​for each unmanned aerial vehicle in the formation at each point in time, according to the formation flight position relative to the competitor. While it is difficult to generate control target values ​​for formation flight with conventional technology, in this embodiment, the filming system generates control target values ​​for the formation flight position in a relative positional relationship with the position of the competitor, who is the subject of the filming.

[0056] The multiple unmanned aerial vehicles that make up the formation may have unmanned aerial vehicles with different roles. In this embodiment, there is a leader vehicle as the first unmanned aerial vehicle and a sub vehicle as the second unmanned aerial vehicle. The leader vehicle is an aircraft that determines the position of the competitor and follows the competitor so that its relative position / relative distance to the competitor's position falls within a target range. The sub vehicle is an aircraft that determines the position of the leader vehicle and follows the leader so that its relative position / relative distance to the leader's position falls within a target range. In the formation, at least one unmanned aerial vehicle is the leader vehicle, and the vehicles other than the leader vehicle are sub vehicle. All vehicles in the formation may be leader vehicles. In addition, either the first tracking control or the second tracking control described below may be applied.

[0057] 3-1. First Tracking Control: First tracking control controls the formation flight positions of multiple unmanned aerial vehicles based on their relative position relative to the position of the competitor, and does not need to align with the direction of the competitor's movement. At least one leader unmanned aerial vehicle (the first unmanned aerial vehicle) among the multiple unmanned aerial vehicles in the formation acquires the position information of the competitor and controls its own position relative to the position of the competitor within a target range. The filming system generates a target range for the leader vehicle as a control target. A sub-vehicle (the second unmanned aerial vehicle other than the leader vehicle) among the multiple unmanned aerial vehicles in the formation acquires the position information of the leader vehicle and controls its own position relative to the position of the leader vehicle within a target range. The filming system generates a target range for the sub-vehicle as a control target. When all vehicles in the formation are leader vehicles, each leader vehicle acquires the position information of the competitor and controls its own position relative to the position of the competitor within a target range. The filming system generates a target range for each leader vehicle as a control target.

[0058] 3-2. Second Tracking Control: Second tracking control controls the formation flight positions of multiple unmanned aerial vehicles relative to the position and direction of movement of the competitor, and changes the formation flight position so that the entire formation flight position rotates to match the direction of movement of the competitor. At least one first unmanned aerial vehicle, the leader vehicle, among the multiple unmanned aerial vehicles in the formation acquires position information and direction of movement information of the competitor and controls its own position relative to the position and direction of movement of the competitor within a target range. The filming system generates a target range for the leader vehicle as a control target. A second sub-vehicle, a sub-vehicle other than the leader vehicle, among the multiple unmanned aerial vehicles in the formation acquires position information and direction of movement information of the leader vehicle and controls its own position relative to the position and direction of movement of the leader vehicle within a target range. The filming system generates a target range for the sub-vehicle as a control target. When all vehicles in the formation are leader vehicles, each leader vehicle acquires position information and direction of movement information of the competitor and controls its own position relative to the position and direction of movement of the competitor within a target range. The imaging system generates a target range for each leader as a control target.

[0059] 4. Dealing with Abnormalities: Next, if an abnormality or malfunction occurs in an unmanned aerial vehicle that is part of the formation, appropriate countermeasures are required. If the filming system determines that there is a formation flight abnormality, it controls the following evacuation action as an appropriate countermeasure. The evacuation action is performed to ensure safety. The filming system determines which unmanned aerial vehicles to evacuate depending on the situation of the formation.

[0060] 4A. Regarding Determining Evacuation Targets: 4A-1. When the Leader Aircraft is Abnormal: If the photography system determines that the leader aircraft is flying abnormally due to a risk of contact, it will determine that the entire formation, including the leader aircraft, is a target for evacuation and will cause the entire formation to take evacuation action.

[0061] 4A-2. When a secondary aircraft is abnormal: If the imaging system determines that a secondary aircraft is abnormal in formation flight due to a collision risk, it will determine at least that secondary aircraft as an evacuation target and cause evacuation action to be taken. In this case, other aircraft in the formation that do not evacuate will continue to capture images. Alternatively, the imaging system may determine the entire formation, including that secondary aircraft, as an evacuation target and cause evacuation action to be taken for the entire formation.

[0062] 4B. Regarding Evasion Actions: Evasion actions are provided according to the formation flight position in the formation. The basic concept of an evacuation action is movement away from the contestant who is the subject of photography. Evasion actions may particularly include movement to increase altitude and ascend. The formation flight positions of the multiple unmanned aerial vehicles in the formation include at least two of the following: a forward position in front of the contestant, a rear position in rear of the contestant, a right position in the right direction of the contestant, a left position in the left direction of the contestant, and a directly above position in the direction directly above the contestant.

[0063] Depending on the formation flight position, the escape action may include the following movements: For unmanned aerial vehicles in the forward position, this may include ascending, moving left or right, or moving backward. For unmanned aerial vehicles in the aft position, this may include ascending, moving left or right, or moving backward. For unmanned aerial vehicles in the right position, this may include ascending, moving right or moving backward. For unmanned aerial vehicles in the left position, this may include ascending, moving left or moving backward. For unmanned aerial vehicles in the direct overhead position, this may include ascending, moving left or moving backward.

[0064] <First Embodiment> An imaging system and the like according to a first embodiment of the present disclosure will be described using Figures 1 and subsequent figures. The imaging system of the first embodiment is a system in which, in a tracking imaging mode, multiple drones (unmanned aerial vehicles) 1 constituting a formation 10 fly so as to follow a competitor P and capture images with a camera, as shown in Figures 1, 36, etc. The imaging method of the first embodiment is a method having steps executed by the server 30 of the imaging system of the first embodiment and the drones 1. If an abnormality is detected in the drones 1, for example, if it is determined that the leader drone has a formation flight abnormality, the imaging system evacuates the formation 10 in an abnormality evacuation mode. The evacuation is, for example, an upward movement to increase altitude as a movement away from the competitor P.

[0065] [Regarding the entity responsible for control processing] In the first embodiment, the entity responsible for the main control processing is described as the server 30 in Fig. 1, but in a modified example, the main entity may be each unmanned aerial vehicle 1. In the first embodiment, the server 30 grasps the status of each drone 1 based on communication with each drone 1 in the formation 10, and determines the control processing in consideration of the overall situation of the formation 10. For example, the server 30 determines which drone 1 to evacuate depending on which drone 1 is experiencing formation flight abnormalities, and controls the evacuation action.

[0066] In a modified example, each drone 1 in the formation 10 is equipped with a determination algorithm for control processing, and while communicating with each other via wireless communication, each drone makes its own independent determination and determines the control processing. For example, a drone 1 grasps the status of the other drones 1 based on communication with the other drones 1 and determines the control processing in consideration of the overall situation of the formation 10. For example, the leader drone 1 may determine which drone 1 to evacuate based on which other drone 1 is experiencing formation flight abnormalities, and control the evacuation action.

[0067] [Regarding Automatic Flight and Manual Flight] In the first embodiment, the formation 10 is described as flying automatically based on a preset setting. However, this is not limiting, and in a modified example, the pilot U1 may fly at least some of the drones 1 in the formation 10 manually. The filming system provides the pilot device 20 of the pilot U1 with support information for manual flight, such as control target values, the status of the drones 1 such as abnormalities, evacuation instructions, etc.

[0068] [Regarding the leader aircraft and sub-aircraft] In the first embodiment, as shown in FIG. 3 etc., the leader aircraft is an aircraft that grasps its relative positional relationship with the athlete P who is the subject of photography and flies while following the athlete P who is the subject of photography. The sub-aircraft is an aircraft that grasps its relative positional relationship with the leader aircraft and flies while following the leader aircraft. At least one leader aircraft is provided in the formation 10. All aircraft in the formation 10 may be leader aircraft (FIG. 9). Note that if it seems strange to have multiple "leader aircraft" in the formation 10, the "leader aircraft" may be renamed to another name, such as main or primary. The leader aircraft may also be referred to as the first unmanned aerial vehicle, the sub-aircraft as the second unmanned aerial vehicle, etc.

[0069] [Entire System] FIG. 1 shows a diagram of the overall system configuration. The system in FIG. 1 may be referred to as a photography system, a drone photography system, a competition photography system, etc. The photography system in FIG. 1 has an athlete P (i.e., athlete, player, etc.) competing in a competition (i.e., a race, sport, etc.) in a competition environment 60 (i.e., a site, etc.). The photography system in FIG. 1 is a system that includes a formation 10 (i.e., a group of aircraft, a group of drones, etc.) of multiple unmanned aerial vehicles 1 (i.e., drones 1, etc.) that follow and photograph the athlete P. The photography system in FIG. 1 has a computer system 100 that is connected to the formation 10 in the competition environment 60 via communications, including wireless communications.

[0070] The computer system 100 includes a controller device 20 used by a controller U1, a business operator's server 30, a presetting terminal 40, and a captured image display device 50, which are connected via a communication network 90. ​​The controller device 20, server 30, presetting terminal 40, captured image display device 50, and other components are connected via communication to the communication network 90. ​​Although not shown, other devices such as a database server and a business operator's management terminal may also be connected. In this example, the controller device 20 includes a controller device 21 and a monitor device 22.

[0071] The competition environment 60 is a competition venue or the like, and includes a competition course 61. An example of the competition is snowboarding, which is shown diagrammatically in the drawings. FIG. 1 shows a state in which an athlete P is on the course 61 during a competition. FIG. 1 also shows a state in which a formation 10 of multiple drones 1, five in this example, is flying near the athlete P and photographing the athlete P from the air. The formation 10 follows the athlete P during the competition and takes aerial photographs.

[0072] In other words, the athlete P is a subject to be photographed, an object to be photographed, or a target object. The athlete P is wearing the athlete terminal 70. The object to be photographed includes not only the athlete P but also an object that moves with the athlete P. Examples of such an object include equipment such as a board, and vehicles such as a bicycle or a yacht.

[0073] The imaging system of FIG. 1 also utilizes known satellites 81, wireless base stations 82, and the like. The satellites 81 transmit signals via the Global Navigation Satellite System (GNSS). The wireless base stations 82 communicate wirelessly with the drones 1 and the like. The drones 1, the athlete terminals 70, and the wireless base stations 82 perform positioning and other operations based on the signals received from the satellites 81. The drones 1 determine their positioning information, such as the date and time, their own position in space, e.g., latitude, longitude, and altitude, through wireless communication with the satellites 81 and the wireless base stations 82. Note that the means of positioning is not limited to satellites 81, and if a positioning system is installed in the competition environment 60, that positioning system may also be used.

[0074] The control device 21 and monitor device 22, which are the pilot device 20 (in other words, the pilot system), are devices used by the user U1, who is the pilot U1. The pilot U1 is a person who pilots or monitors the drone 1 by operating the control device 21 and monitor device 22. The control device 21 and monitor device 22 communicate with each other. The control device 21 pilots the drone 1 based on operations and settings by the pilot U1. In other words, the control device 21 can instruct the flight of the drone 1. The monitor device 22 has a display screen, and displays a monitor screen for piloting the drone 1 on the display screen when piloting the drone 1 based on video data from the control device 21. The monitor image displayed on the monitor device 22 is an image based on video data, etc. transmitted from the drone 1.

[0075] The control device 21 and monitor device 22 communicate wirelessly with the drone 1 on-site, the competitor terminal 70, the wireless base station 82, and the like. The drone 1 can also be automatically controlled based on the pre-settings made on the pre-setting terminal 40. In the case of manual control, the pilot U1 manually controls the control device 21 while watching the screen of the monitor device 22, thereby manually controlling the drone 1. The control device 21 and monitor device 22 may also be configured as an integrated device.

[0076] The pre-setting terminal 40 is a device for inputting pre-settings related to the flight of the drone 1. The pilot U1 inputs pre-settings into the pre-setting terminal 40. The pre-setting terminal 40 works in conjunction with the server 30. The input information at the pre-setting terminal 40 is sent to the server 30, where pre-setting processing is performed and the pre-setting information is saved. The pre-setting terminal 40 and the server 30 may be integrated into one unit. The pre-setting terminal 40 and the pilot device 20 may be integrated into one unit.

[0077] The captured video display device 50 receives video of the contestant P captured by the drone 1's camera (see FIG. 2 ), which will be described later, and displays it on its display screen. Specifically, the drone 1 transmits video data 250 of the video captured by the camera to the pilot device 20 via wireless communication. The pilot device 20 then transmits the video data 250 to the server 30 via communication. The server 30 then processes the video data 250 and transmits the resulting video data 250 to the captured video display device 50. The captured video display device 50 displays the captured video on its display screen based on the received video data 250. For example, a user U2, such as a viewer or a race commentator, can view the video on the captured video display device 50. The pilot device 20 or the server 30 may perform predetermined processing on the video data 250 from the drone 1 to create distribution video data suitable for viewing. In a modified example, the video data 250 from the drone 1 may be transmitted to the server 30 or the captured video display device 50.

[0078] The server 30 communicates with the drones 1 of the formation 10 via the pilot device 20 and performs processing related to predetermined functions characteristic of the embodiment. The processing of the predetermined functions includes control of the flight of the formation 10 and control of photography. The processing of the predetermined functions may be performed mainly by the server 30, or may be performed mainly by the drones 1, or may be performed partially by the pilot device 20, etc. The processing of the predetermined functions may be realized by sharing the work among the drones 1, the server 30, the pilot device 20, etc.

[0079] [Competitions, Races, Sports] The competitions, races, and sports targeted in this embodiment are, in particular, competitions with high speeds and intense movement, competitions held on unpaved courses 61, and in particular competitions held by athletes P alone. Examples of such competitions include snowboarding, skiing, yachting, surfing, car racing, motorcycling, cycling, and marathons. These competitions are targets for which tracking and aerial photography by the formation 10 of drones 1 are effective.

[0080] [Formation] Figure 2 shows an example of the formation flight positions of a formation 10 of multiple drones 1 relative to a competitor P, who is the subject of photography. Figure 2 is Example 1. In this example, the formation 10 has five drones 1, drones 1a, 1b, 1c, 1d, and 1e. Figure 2 also shows the cameras {Ca, Cb, Cc, Cc, Cd, Ce} of each drone 1. Drone 1a is equipped with camera Ca, drone 1b is equipped with camera Cb, drone 1c is equipped with camera Cc, drone 1d is equipped with camera Cd, and drone 1e is equipped with camera Ce. In addition, a competitor P is wearing a competitor terminal 70.

[0081] The movement of the drones 1 in formation 10 following the athlete P is based on the athlete P's movement direction PD1 or the direction of travel of the course 61. Figure 2 shows a course portion 61a of the course 61. The direction and position of the drones 1 in formation 10, such as forward, backward, left, and right, are determined relative to the athlete P's position and movement direction PD1. In the example of Figure 2, the front is in the same direction as the movement direction PD1, and the rear is in the opposite direction. Furthermore, left and right and up and down are determined relative to the movement direction PD1. In the example of Figure 2, when the spatial coordinate system of the competition environment 60 is expressed as (X, Y, Z), the direction of travel of the course portion 61a is the positive Y direction (+Y). Note that the positive Y direction is also referred to as +Y, and the negative Y direction is also referred to as -Y. The same applies to other directions.

[0082] In the state shown in Figure 2, drone 1a is placed diagonally above and ahead of athlete P as the front position, and drone 1b is placed diagonally above and behind as the rear position. A diagonally above position refers to a position higher than the height of athlete P relative to the road surface of course 61. Furthermore, drone 1c is placed diagonally above and to the right of athlete P as the right position, and drone 1d is placed diagonally above and to the left as the left position. Furthermore, in this example, drone 1e is placed directly above athlete P in the vertical direction. These positions are also shown in Figures 3 and 4, which will be described later.

[0083] In this example, the front and rear positions are diagonally upward positions, which is to reduce the possibility of obstructing the movement of the athlete P. Alternatively, this is to increase the visibility of the athlete P. Also, in this example, the left and right positions are diagonally upward positions, but this is not limiting and they may be positions at the same height as the athlete P ( FIG. 6 ). This is because the left and right positions are less likely to obstruct the movement of the athlete P. In the embodiment, the front position includes the concept of a diagonally upward position in front, the rear position includes the concept of a diagonally upward position behind, the right position includes the concept of a diagonally upward position to the right, and the left position includes the concept of a diagonally upward position to the left. Diagonal means having an angle in the vertical direction with respect to the road surface.

[0084] The positions of each drone 1 can be restated as follows in terms of the camera's shooting direction. In this example, the positive Y direction (+Y) is taken as the front, based on the reference. For camera Ca of the front drone 1a, the shooting direction is facing diagonally downward and rearward. For camera Cb of the rear drone 1b, the shooting direction is facing diagonally downward and forward. For camera Cc of the right drone 1c, the shooting direction is facing diagonally downward and left. For camera Cd of the left drone 1d, the shooting direction is facing diagonally downward and right. For camera Ce of drone 1e directly above, the shooting direction is facing directly downward.

[0085] The image captured by camera Ca has an angle of view that captures the athlete P from the front in the direction of movement PD1. The image captured by camera Cb has an angle of view that captures the athlete P from behind. The image captured by camera Cc has an angle of view that captures the athlete P from the right side. The image captured by camera Cd has an angle of view that captures the athlete P from the left side. The image captured by camera Ce has an angle of view that captures the athlete P and others from directly above in a bird's-eye view (see FIG. 25, described below).

[0086] Figure 3 is an X-Y plan view of Example 1 of the formation flight position of the formation 10 in Figure 2, as viewed from above in the vertical direction (+Z). The direction of movement PD1 of the competitor P is the Y direction (+Y). Drone 1a is positioned a predetermined distance d1 in front of the competitor P's position PP1 in the Y direction, and drone 1b is positioned a predetermined distance d2 behind the competitor P's position PP1 in the Y direction (-Y). Drone 1c is positioned a predetermined distance d3 to the right of the competitor P's position PP1 in the X direction (+X), and drone 1d is positioned a predetermined distance d4 to the left of the competitor P's position PP1 in the X direction (-X). Drone 1e is positioned a predetermined distance above the competitor P's position PP1 in the Z direction (+Z).

[0087] Each drone 1 in the formation 10 is assigned a number, ID, or other identification information for management purposes. This information may be any information that allows identification. The photography system manages the identification of each drone 1.

[0088] Figure 4 is a Y-Z plane view of Example 1 of the formation flight position of the formation 10 in Figures 2 and 3, as seen from the right side (+X) in the horizontal direction. The direction of movement PD1 of the athlete P is slightly diagonally downward in the Y direction (+Y). In this example, the course portion 61a is downhill, so the direction of movement PD1 is diagonally downward to match the descent. Drones 1a, 1b, 1c, and 1d are positioned at height h1 in the height direction (in this example, perpendicular to the road surface) from the position of the road surface of the course portion 61a or the height position of the athlete P. Drone 1e is positioned at a higher height h2 (h2 > h1) from the position of the road surface of the course portion 61a or the height position of the athlete P.

[0089] 4, the position of each drone 1 in the formation 10 is selected to be parallel to the slope of the road surface of the course 61. However, if the road surface of the course 61 is sloped from the horizontal, the position of each drone 1 in the formation 10 may be selected to be horizontal or vertical to the competitor P, rather than parallel to the road surface.

[0090] The definition of direction and position is not limited to the above example. In the example of FIG. 2 , in a spatial coordinate system (X, Y, Z) corresponding to the competition environment 60, the direction of travel of a certain course portion 61a of a snowboarding competition course 61 is the illustrated Y direction (+Y), and the direction of travel of the athlete P is the movement direction PD1. In this case, the tracking movement direction AD1 of the formation 10 in this course portion 61a is also roughly the Y direction (+Y). During this tracking movement, the orientation of each camera is roughly maintained as shown in the figure. Of course, some shaking and other factors occur during flight, which may cause the position and orientation to fluctuate, but predetermined tracking control is performed, so the position and orientation are brought closer to a predetermined target state.

[0091] For example, a course 61 in a snowboarding competition may have several turns to the left and right relative to the direction of travel. The formation 10 and cameras may follow each turn roughly (see FIG. 8B, described below). For each section of the course, the images captured by each camera are stabilized so that their orientation remains roughly constant. This makes it easier to view the competition objectively in the captured images.

[0092] In another example, the direction of the formation 10 may be defined based on the spatial coordinate system of the competition environment 60. If the positional relationship of the formation flight positions of the multiple drones 1 in the formation 10 is appropriate, they may not be aligned with the direction of movement of the athlete P (see FIG. 8A described below). For example, if a snowboarding course 61 has multiple course sections divided according to turns, the orientation of the formation 10 and the camera may be maintained in roughly the same direction across the multiple course sections.

[0093] In another example, when the direction of the course 61 or the direction of the competitor P fluctuates in detail, the movement direction of the formation 10 and the camera may be changed in more detail accordingly. For example, even within one course portion 61a in Figure 2, when the movement direction of the competitor P fluctuates by swaying from side to side, the direction of the formation 10 and the camera may be changed accordingly.

[0094] The above-mentioned regulations regarding the direction and position of the drone 1 and the camera can be selected and set according to the characteristics of the competition, etc., such as the formation flight position setting unit 316 in Figure 18A described later.

[0095] [Leader and Sub-Drone in Formation] Each of the multiple drones 1 in the formation 10 may be assigned a role, such as a leader or sub-drone, as described below. As an example, in the formation 10 of FIGS. 2 to 4, the drone 1a in the front position is set as the leader drone, and the other drones 1, drones 1b, 1c, 1d, and 1e, are set as sub-drone. That is, in a first example, only one drone in the formation 10 is set as the leader drone. In another example, drone 1e may be set as the leader drone. Various settings are possible for the leader drone and the sub-drone. For example, see the leader / sub-drone setting unit 317 in FIG. 18A, which will be described later.

[0096] The leader drone 1a tracks its own position based on the position and direction of movement PD1 of the contestant P. The sub drones 1b, 1c, 1d, and 1e track their own positions based on the position and direction of movement of the leader drone 1a.

[0097] Figure 5 shows the positional relationship between the leader drone and the sub drones in Figure 3 and tracking control. The leader drone 1a determines the position, movement direction PD1, speed, etc. of the athlete P based on positioning and detection, and determines a forward position La as shown in the figure as its own position relative to the athlete P's position, etc. The dashed circle indicates the target range for the forward position La. Drone 1a determines its own forward position La as a position that is a predetermined distance d1 (Figure 3) forward in the Y direction relative to the athlete P's position PP1, in line with the movement direction PD1.

[0098] Furthermore, the sub-drone, for example drone 1b, grasps the position, direction of movement, speed, etc. of the leader drone 1a based on positioning and detection, and determines its own position relative to the leader drone's position, etc., as the illustrated rear position Lb. Drone 1b determines its own rear position Lb as a position that is a predetermined distance (d1 + d2 in Figure 3) in the rear direction (-Y) from the position of drone 1a.

[0099] Furthermore, a sub-drone, for example, drone 1c, determines the position, direction of movement, speed, etc. of the leader drone 1a based on positioning and detection, and determines its own position relative to the leader drone's position, etc., as shown in the figure as right position Lc. The same applies to drone 1d in left position Ld and drone 1e directly above. In Figure 5, their relative relationships are indicated by dashed arrows.

[0100] For example, when the conditions regarding the positional relationship between the leader drone and the sub drone are met, as shown in Figure 5, the multiple drones 1 in the formation 10 can follow the competitor P and photograph the competitor P while maintaining a controlled, predetermined formation flight position, as shown in Figures 2 to 4. When the predetermined positional relationship between the leader drone and the sub drone is met, a positional relationship that ensures sufficient distance between each drone 1 and the competitor P is also necessarily met.

[0101] In the examples of Figures 3 and 5, the drone 1a in the front position is at about the same distance from the drone 1b in the rear position to the contestant P, but this is not limited to this, and the distance between each drone 1 and the contestant P may be different. For example, the distance between the drone 1a in the front position and the contestant P may be the closest. Since it is desirable for the leader drone 1 to be able to accurately grasp and follow the position of the contestant P, the position of the leader drone 1 may be the closest to the contestant P.

[0102] [Formation Flight Position: Second Example] Figure 6 shows an example of the position of the formation 10, which is a modification of Figure 4. Figure 6 is a second example. For example, drones 1c and 1d on the left and right may be positioned at a height h3, or in other words, at an altitude, approximately the same as the height of athlete P, as shown in the figure. In the example of Figure 6, drone 1e directly above is not provided, and the formation 10 is made up of four drones 1 (1a to 1d). Depending on the competition, if there is no other athlete following athlete P, drone 1b in the rear position may also be positioned at a height h3 approximately the same as the height of athlete P, as it is unlikely to interfere with athlete P.

[0103] [Formation Flight Positions: Third Example] Figure 7 shows an example of the positions of drones 1 in a formation 10 in another modified example. Figure 7 is a third example. In this example, drone 1f is positioned diagonally forward to the right, drone 1g is positioned diagonally forward to the left, drone 1h is positioned diagonally behind the right, and drone 1i is positioned diagonally behind the left, based on the position of athlete P, within a plane corresponding to the road surface. For example, drone 1f captures images using camera Cf at an angle of view that captures a diagonal view to the right of athlete P. Depending on the competition, drone 1 may be positioned diagonally within a plane corresponding to the road surface, avoiding positions in front, behind, left, or right from the perspective of athlete P. In addition to the first to third formation flight positions described above, formation flight positions may also be used in which multiple drones fly in the same direction relative to the athlete but at different altitudes.

[0104] [Two types of tracking control] Formation flight control in the tracking photography mode described below, in other words, tracking control, can be performed in either of two ways: controlling only the relative positional relationship with respect to the position of the athlete P, or controlling the relative positional relationship that is in accordance with the position of the athlete P as well as the direction of movement of the athlete P, as described below, and can be selectively applied.

[0105] [First Tracking Control] Figure 8A shows the first tracking control in which control is performed only based on the relative positional relationship with respect to the athlete P. The first tracking control in Figure 8A controls the formation flight positions of the multiple drones 1 in the formation 10 based on the relative positional relationship with respect to the position of the athlete P, without tracking the direction of movement of the athlete P.

[0106] On course 61, for example, the position of contestant P at a certain point in time is position PPa, and the direction of movement is direction PDa. At this time, the leader drone, for example drone 1a, determines its own forward position La, maintaining a predetermined relative distance in a predetermined direction, e.g., the +Y direction, from position PPa. The other sub-drones, drones 1b, 1c, and 1d, each determine their own positions in a predetermined positional relationship relative to the leader drone 1a's position La. For example, drone 1b determines its rear position Lb, maintaining a predetermined relative distance in the -Y direction. Drone 1c determines its right position Lc, maintaining a predetermined relative distance in the +X direction. Drone 1d determines its left position Ld, maintaining a predetermined relative distance in the -X direction. As a result, the four drones follow the position of contestant P in formation flight positions with predetermined positional relationships, such as +Y, -Y, +X, and -X, i.e., front-to-back, left-to-right positional relationships. In the control of FIG. 8A, the camera orientation is maintained approximately constant within the spatial coordinate system.

[0107] 8B shows a case where the second tracking control is performed in accordance with the position and movement direction of the athlete P. If the movement direction of the athlete P changes during the competition, the imaging system also changes the position of each aircraft in the formation 10 to match the change in the movement direction.

[0108] The leader drone, for example drone 1a, controls its own forward position La and camera orientation to achieve a predetermined relative positional relationship, for example, in accordance with the position PPa and movement direction PDa of contestant P. Each sub-drone in the formation 10 controls its own position and camera orientation to achieve a predetermined relative positional relationship, in accordance with the position and movement direction of the leader drone. Furthermore, when contestant P is moving in the movement direction PDb, drone 1a is positioned at a forward position La that matches that movement direction PDb. In the case of the control in Figure 8B, the camera orientation is such that contestant P is captured at a predetermined angle of view.

[0109] It should be noted that the position of the formation 10 does not have to be limited to perfectly tracking changes in the direction of movement of the subject being photographed. If there is a large fluctuation in the direction of movement of the subject being photographed, the direction of the formation 10 may be tracked more gradually with smaller fluctuations. Furthermore, if there is a large fluctuation in the direction of movement of the subject being photographed (if the subject turns 180 degrees), moving each aircraft in the formation 10 in a straight line to the target position may result in a collision due to the crossing of the movement path with other aircraft. Therefore, if there is a large fluctuation in the movement direction of the subject being photographed by a predetermined value or more, each aircraft can be moved in an arc around the athlete to prevent the movement path from crossing the movement path of other aircraft.

[0110] The filming system generates a control target value at each timing for controlling the formation flight to follow the contestant P as described above. Each aircraft 1 controls its flight so as to approach the control target value. The control target value may be a target range for the position of the drone 1 as shown in Figure 5 above.

[0111] [Leader and Sub-Vehicles: Second Example] Figure 9 shows another example of the leader and sub-vehicle configurations for the formation 10 shown in Figure 2. In the second example shown in Figure 9, all five drones 1 (1a-1e) in the formation 10 are configured as leader drones, with no sub-vehicles. Each drone 1 determines the position, direction of movement, speed, etc. of the competitor P based on positioning and detection, and determines its own position relative to the competitor P's position. As a result, for example, as shown in Figure 9, when the conditions for the positional relationship between each leader drone and the competitor P are met, the multiple drones 1 in the formation 10 can follow and photograph the competitor P while maintaining a controlled, predetermined formation flight position, as in Figure 2. When the predetermined positional relationship between each leader drone and the competitor P is met, a positional relationship that ensures sufficient distance between each drone 1 is also necessarily met.

[0112] The two types of tracking control shown in Figures 8A and 8B can be applied whether one part of the formation 10 is the leader aircraft as in Figure 3 or all aircraft are the leader aircraft as in Figure 9.

[0113] [Additional Function: Determination by Sub-Aircraft] Figure 10 shows an example of a case where each drone 1 in the formation 10, even if it is a sub-aircraft, has an additional function of checking the relative distance, etc., to ensure safety between it and other drones 1 and the contestant P. The sub-aircraft may check the relative distance and relative speed between it and the contestant P and other sub-aircraft, and if it is outside the target range, it may determine that there is an abnormality in the formation flight.

[0114] In the example of Figure 10, only drone 1a is the leader drone, similar to Figures 3 and 5. Regardless of whether it is the leader drone or a sub-drone, each drone 1 checks to ensure that the distance between itself and competitor P is at least a predetermined distance, and also checks to ensure that the distance between itself and other drones is at least a predetermined distance.

[0115] In the example of Figure 10, the explanation will focus on drone 1c in the right position. Drone 1c, which is the sub-drone, determines its own right position relative to the position of drone 1a, which is the leader drone, in the front position. Drone 1c controls itself so that the distance dca between drone 1a and drone 1a is equal to or greater than a predetermined distance (the lower limit of the target range). Drone 1c also checks whether the distance dcp between drone 1c and contestant P is equal to or greater than the predetermined distance. Drone 1c also checks whether the distance dcb between drone 1b, which is in the rear position, is equal to or greater than the predetermined distance. Drone 1c also checks whether the distance dcd between drone 1d, which is in the left position, is equal to or greater than the predetermined distance. Drone 1c also checks whether the distance dce between drone 1e directly above drone 1c is equal to or greater than the predetermined distance. These checks may be performed, for example, by using a sensor provided on drone 1 to detect the distance between drone 1 and other drones 1 and determine whether the distance is equal to or greater than the predetermined distance.

[0116] If the distance between each drone 1 and the athlete P or between each drone 1 and another drone 1 is less than a predetermined distance, the drone 1 determines that the formation flight is abnormal due to the risk of collision. Each of the predetermined distances described above can be a control setting. Similarly, each drone 1 may determine that the formation flight is abnormal if the relative speed between the drone 1 and the athlete P or another drone 1 does not satisfy a predetermined condition.

[0117] [Unmanned Aerial Vehicle (Drone)] Figure 11 shows an example of the functional block configuration of the unmanned aerial vehicle 1, which is the drone 1 of Figure 1. The unmanned aerial vehicle 1 has a photographing unit 110, a flight function unit 120, an obstacle detection unit 130, a flight status measurement unit 140, a reception status detection unit 160, and a communication unit 170. The unmanned aerial vehicle 100 also has a processor 1001, a memory 1002, a battery 1003, etc. These components are interconnected via an architecture such as a bus, enabling communication, input / output, etc.

[0118] The processor 1001 reads and writes data and information from the memory 1002 and realizes predetermined functions through program processing. The functions are not limited to software program processing, but can also be implemented using hardware circuits such as FPGAs. Although each unit, such as the imaging unit 110, is illustrated separately from the processor 1001, it may also be realized using processing by the processor 1001.

[0119] The photographing unit 110 has a photographing camera (camera) 111, a camera holding unit 112, a camera control unit 113, a holding control unit 113, a camera state acquisition unit 115, a camera abnormality detection unit 116, and an image processing unit 117. The photographing camera (camera) 111 corresponds to, for example, camera Ca in Fig. 2. The camera holding unit 112 is a drive mechanism, a so-called gimbal, that stably holds the camera 111 so that it can move.

[0120] The camera control unit 113 controls parameter values ​​such as the zoom amount, shutter speed, and camera orientation of the camera 111. The holding control unit 114 controls the driving of the camera holding unit 112 to hold the camera 111. The orientation of the camera 111 can be defined as the orientation around each of three orthogonal axes (X, Y, Z).

[0121] The camera state acquisition unit 115 acquires parameter values ​​such as the zoom amount and camera direction of the camera 111. The camera abnormality detection unit 116 detects a state in which the camera 111 is not able to take pictures normally, such as a cloudy lens, as an abnormality.

[0122] The image processing unit 117 detects the object being photographed in the image captured by the camera 111, and adjusts parameter values ​​such as the zoom amount of the camera 111 so that the size and sense of distance of the object being photographed within the angle of view of the image remain as constant as possible. The image processing unit 117 also detects the object being photographed in the image captured by the camera 111, and adjusts parameter values ​​such as the camera direction so that the object being photographed is located as close to the center as possible within the angle of view of the image. The image processing unit 117 may also detect the object being photographed in the image captured by the camera 111, and estimate the position, speed, movement direction, etc. of the object being photographed within space.

[0123] The flight function unit 120 has a motor 121, a propeller 122, a flight control unit 123, and a flight abnormality detection unit 124. The motor 121 drives the propeller 122. The flight control unit 123 controls the motor 121 based on control commands from the pilot device 21. This controls flight using the propeller 122. The flight abnormality detection unit 124 detects abnormalities in the flight function from the behavior of the motor 121, etc.

[0124] The obstacle detection unit 130 includes an obstacle detection sensor 131 and a detection processing unit 132. The obstacle detection sensor 131 is a type of sensor for detecting obstacles for the drone 1 and can be realized, for example, by an optical camera, a ToF (Time of Flight) sensor, a LiDAR (Light Detection and Ranging), or an IR (Infrared) sensor. The detection processing unit 132 detects the athlete P, other drones 1, and other obstacles based on sensing information from the obstacle detection sensor 131. The photography system may calculate and detect the positions and movement directions of a certain unmanned aerial vehicle (also referred to as the athlete's own aircraft) based on information about the position and movement direction of the athlete P or other aircraft, and information about the distance between the athlete P and other aircraft detected by the obstacle detection sensor 131, etc.

[0125] The flight condition measurement unit 140 has an aircraft position measurement unit 141, an aircraft orientation measurement unit 142, and an aircraft speed measurement unit 143. The flight condition measurement unit 140 measures the flight condition of the aircraft itself. The aircraft position measurement unit 141 measures the position of the aircraft itself in space based on signals from satellites 81 and communications with wireless base stations 82. The aircraft orientation measurement unit 142 measures the orientation of the aircraft itself. The aircraft speed measurement unit 143 measures the speed of the aircraft itself.

[0126] The reception status detection unit 160 has a control communication abnormality detection unit 161 and a positioning status abnormality detection unit 162. The control communication abnormality detection unit 161 detects abnormalities in control communication. If an abnormality occurs in the control communication of a leader aircraft or a sub-aircraft in the formation 10, the control communication abnormality detection unit 161 determines that there is an abnormality in the formation flight control of the leader aircraft or the sub-aircraft. An abnormality in control communication occurs when communication is interrupted or delayed, for example.

[0127] The positioning status abnormality detection unit 162 detects an abnormality in the positioning of the unmanned aerial vehicle 1. If the positioning accuracy of the position of the leader aircraft or sub aircraft in the formation 10 deteriorates, the positioning status abnormality detection unit 162 determines that there is an abnormality in the formation flight control of the leader aircraft or sub aircraft. The positioning status abnormality detection unit 162 also detects an abnormality in the positioning of the athlete P who is the subject of the photograph. If the positioning accuracy of the position of the athlete P deteriorates, the positioning status abnormality detection unit 162 determines that there is an abnormality in the formation flight control of the leader aircraft. Examples of cases where the positioning accuracy has deteriorated include weak GNSS signal strength from satellite 81 and multipath of satellite signals.

[0128] The communication unit 170 is a part in which various communication interfaces are implemented, such as a communication interface with the flight control device 21, a communication interface with other aircraft 100, and a communication interface with the competitor terminal 70.

[0129] [Appearance of Drone] Fig. 12 shows an example of the configuration of the appearance of the drone 1. The drone 1 has a camera 111 held on the underside of the main body 1000 via a camera holding unit 112. An obstacle detection sensor 131 is installed on the front of the main body 1000. A plurality of motors 121 and propellers 122 are provided on the side of the main body 1000.

[0130] The obstacle detection sensor 131 can be realized by, for example, an optical camera, a ToF sensor, a LiDAR, an IR sensor, etc. The obstacle detection sensor 131 may also be configured as a stereo camera with two optical cameras provided facing forward of the body of the drone 1. The obstacle detection sensor 131 may not only be a sensor that detects the forward direction provided on the front surface of the main body 1000, but also a sensor that can detect in directions other than the forward direction, such as provided on all six surfaces of the main body 1000.

[0131] A thrust generating device is configured by the propeller 122 and the motor 121. The thrust generating devices may be provided as four thrust generating devices at four locations on the aircraft as in the illustrated example, or may be provided as multiple thrust generating devices at multiple locations.

[0132] [Control device 21] Figure 13 shows an example of the functional block configuration of the control device 21 in Figure 1. The control device 21 has a display unit 210, an input unit 220, a communication unit 230, etc. The control device 21 also has a processor 2001, a memory 2001, a battery 2003, etc. Functions may be realized by program processing by the processor 2001.

[0133] The display unit 210 has an abnormal state display unit 211, a flight state display unit 212, and athlete information display unit 213 (in other words, a photographing target information display unit). Note that the display unit 210 may be not only a display provided in the control device 21, but also the monitor device 22 in Figure 1.

[0134] The input unit 220 includes an aircraft position operation unit 221 , an aircraft attitude operation unit 222 , a camera attitude operation unit 223 , a camera zoom operation unit 224 , a flight command input unit 225 , and a power supply input unit 226 .

[0135] The abnormality status display unit 211 displays the results of the determination made by this photography system regarding abnormalities in the formation flight control and abnormalities in the reception status.

[0136] The flight status display unit 212 displays the flight status of the multiple drones 1 in the formation 10. The flight status displayed may include the flight mode, the relative distance and relative speed to the athlete P, and the relative distance and relative speed between the drones 1. Examples of the flight mode include the tracking shooting mode and emergency evacuation mode shown in FIG. 19 (described later).

[0137] The athlete information display unit 213 displays the captured video of the athlete P and athlete information estimated from the unmanned aerial vehicle control information. Examples of the athlete information include speed and distance traveled.

[0138] The flight command input unit 225 has the following functions. The flight command input unit 225 can switch between manual control and automatic control. The flight command input unit 225 can also input a switch from the tracking photography mode in FIG. 19 to the landing mode. The tracking photography mode is a mode in which the formation 10 tracks and photographs the contestant P. The landing mode is a mode in which the formation 10 lands. In the landing mode, the flight command input unit 225 can also specify landing operations such as landing permission, hovering standby, soft landing on the spot, and propeller stop.

[0139] The communication unit 230 is connected to the unmanned aerial vehicle 1, the athlete terminal 70 which is the athlete P side device, the server 30, the pre-setting terminal 40, and the captured image display device 50 via communication lines, and is equipped with communication interfaces corresponding to them.

[0140] The control device 21 is connected to a monitor device 22 via a communication unit 230. Although not shown, the monitor device 22 is equipped with functions such as a display unit 240 for displaying a monitor screen based on video data 250 from the control device 21.

[0141] [External Appearance of Control Device] Figure 14 shows an example of the external configuration of the control device 21. The control device 21 has a display connected to the main body 2100 as a display unit 210, and control information is displayed on the display. The main body 2100 is provided with a right stick 2101, a left stick 2102, a right lever 2103, a left lever 2104, and the like. For example, as indicated by the arrows, moving the right stick 2101 to the right corresponds to moving the drone 1 to the right, moving it to the left corresponds to moving the drone 1 to the left, moving it up corresponds to moving the drone 1 upward, and moving it down corresponds to moving the drone 1 downward. For example, as indicated by the arrows, moving the left stick 2102 to the right corresponds to turning the drone 1 right, moving it to the left corresponds to turning the drone 1 left, moving it up corresponds to moving the drone 1 forward, and moving it down corresponds to moving the drone 1 backward. The right lever 2103 corresponds to, for example, panning the camera, and the left lever 2104 corresponds to, for example, zooming the camera. An overview of each of the above-mentioned operations related to the drone 1 is shown at the bottom of Figure 14.

[0142] The control device 21 transmits a signal according to the operation to the drone 1. The drone 1 controls its operation according to the received signal.

[0143] 15 shows an example of a functional block configuration of the presetting terminal 40. The presetting terminal 40 includes a display unit 410, an input unit 420, a communication unit 430, etc. The presetting terminal 40 also includes a processor 4001, a memory 4002, a battery 4003, etc.

[0144] The display unit 410 has a setting information display unit 411 and an error display unit 412. The setting information display unit 411 displays information input by each input unit of the input unit 420. If the input formation flight position or standby position (described later) is set to a position that deviates from the safety range set for each event being photographed, the error display unit 412 determines that there is a position setting error and displays the details of the error.

[0145] The input section 420 has a photographed competition information input section 421, a photographed athlete information input section 422, a course / geofence input section 423, a standby position input section 424, a takeoff position / landing position input section 425, a formation flight position input section 426, a leader / sub-aircraft input section 427, an evacuation input section 428, a priority input section 429, etc.

[0146] The photography target competition information input unit 421 inputs information about the competition to be photographed. The photography target athlete information input unit 422 inputs information about the athlete P to be photographed. The course / geofence input unit 423 inputs the competition course 61 and the geofence that matches it. The geofence is information for restricting the space in which the unmanned aerial vehicle 1 can fly (Figure 24). The standby position input unit 424 inputs the standby position, which will be described later (Figure 21, etc.). The takeoff position / landing position input unit 425 inputs the takeoff position and landing position, which will be described later.

[0147] The formation flight position input unit 426 inputs flight positions and conditions of the drones 1 in the formation 10, such as the example shown in Figure 2. For example, in the case of drone 1a, the input information, or in other words, setting information, includes the target position and distance in each axial direction relative to the position of athlete P, and an allowable range (e.g., upper and lower limits) centered on the target position. The axial directions here refer to the longitudinal, lateral, and vertical axes as relative directions and relative coordinate systems. For example, in the example shown in Figure 3, when setting the position of drone 1a, the drone must be at least a distance d1 forward from the position PP1 of athlete P, and in the example shown in Figure 4, the drone must be at least a height h1 in the vertical direction from the position of athlete P.

[0148] The conditions may include judgment conditions for determining the risk of contact between drones 1 or between drone 1 and athlete P.

[0149] The leader / sub aircraft input unit 427 inputs the settings of the leader aircraft and sub aircraft as in the example of Figure 3. At least one aircraft in the formation 10 must be set as the leader aircraft. It is not permitted to set all aircraft in the formation 10 as sub aircraft. All aircraft in the formation 10 may be set as leader aircraft as in the example of Figure 9.

[0150] The evacuation input unit 428 inputs settings related to the evacuation control operation described below. The evacuation input unit 428 can set and input specific movements and conditions for evacuation in a direction away from the contestant P for each drone 1 in the formation 10.

[0151] The priority input unit 429 inputs the priority settings described below. Priorities can be set for formation flight positions. Evacuation control using the priorities is possible.

[0152] The communication unit 430 is connected to the operator device 20, the server 30, the captured image display device 50, and the like via communication lines.

[0153] 16 shows an example of a functional block configuration of the captured image display device 50. The captured image display device 50 includes a display unit 510, an input unit 520, a communication unit 530, etc. The captured image display device 50 also includes a processor 5001, a memory 5002, a battery 5003, etc.

[0154] The display unit 510 has a main camera video display unit 511 and an other camera video display unit 512. The main camera video display unit 511 displays video from the main camera (described later) ( FIG. 25 ). The other camera video display unit 512 displays video from cameras other than the main camera. It is possible to set and switch which of the videos from the multiple cameras in the formation 10 will be used as the main camera.

[0155] The input unit 520 has a camera switching unit 521. The camera switching unit 521 switches between the images from the main camera and other cameras displayed on the display unit 510 based on an operation input from the user U2.

[0156] The communication unit 530 is connected to the operator device 20, the server 30, the setting terminal 40, etc. via communication lines.

[0157] The captured image display device 50 is a device intended for users U2, such as general viewers and race commentators. By operating the camera switching unit 521, the user U2 can view the camera image they want to view by enlarging it as the main camera image (FIG. 25).

[0158] 17 shows an example of the functional block configuration of the athlete terminal 70. The athlete terminal 70 includes a status acquisition unit 710, a photographing unit 720, an abnormality detection unit 730, a communication unit 740, etc. The athlete terminal 70 also includes a processor 7001, a memory 7002, a battery 7003, etc.

[0159] The athlete terminal 70 is mounted on the athlete P or on a vehicle or equipment used by the athlete P. For example, the athlete terminal 70 may be mounted on the athlete P's clothes, hat, helmet, board, bicycle, yacht, etc.

[0160] The status acquisition unit 710 is realized by various sensors, etc. The status acquisition unit 710 includes a position measurement unit 711, a speed measurement unit 712, a movement direction measurement unit 713, etc. The position measurement unit 711 measures the position of the athlete P using GNSS / GPS, etc. The speed measurement unit 712 measures the speed of the athlete P. The movement direction measurement unit 713 measures the movement direction of the athlete P (e.g., movement direction PD1 in FIG. 2 ). The status acquisition unit 710 can acquire the current position, speed, movement direction, etc. using GNSS / GPS, as well as an inertial measurement unit (IMU) or transponder, which can measure angular velocity and acceleration with high accuracy. When using a transponder, a transmitter and reader are installed on the ground and on the athlete P's side, respectively, and the equipment is installed at predetermined intervals on the ground course 61. This makes it possible to detect the athlete P's position, speed, movement direction, etc.

[0161] The photographing unit 720 has an onboard camera 721. The onboard camera 721 photographs an image from the viewpoint of the athlete P looking at the surroundings (for example, in the direction of travel). The photographing unit 720 may not be provided.

[0162] The abnormality detection unit 730 has a positioning state abnormality detection unit 731. The positioning state abnormality detection unit 731 detects an abnormality related to the state acquisition by the state acquisition unit 710, for example, an abnormality in positioning by the position measurement unit 711.

[0163] Additionally, if it is desired to further enhance safety, the athlete terminal 70 may have a function to detect the surrounding drones 1 in the formation 10. For example, the athlete terminal 70 may determine that an abnormality has occurred when the distance to the surrounding drone 1 falls below a predetermined distance. By providing this determination / detection result to the drone 1 or the server 30, it is possible to control measures to reduce the risk of collision.

[0164] In this photography system, the athlete terminal 70 is not essential, but if provided, more advanced control becomes possible.

[0165] [Server] Figures 18A and 18B show an example of the functional block configuration of the server 30. Figure 18A shows the first half, and Figure 18B shows the second half. The server 30 includes a pre-takeoff setting unit 310, an aircraft status information display unit 320, a standby abnormality determination unit 330, a competition information acquisition unit 340, an athlete information acquisition unit 350, a control target command unit 360, an abnormality determination unit 370, an evacuation action determination unit 380, a distribution video control unit 390, a communication unit 395, and the like. The server 30 also includes a processor 3001, a memory 3002, a battery 3003, and the like.

[0166] The pre-takeoff setting unit 310 is a part that performs settings on the server 30 side corresponding to the pre-settings made on the pre-setting terminal 40, and this setting is performed based on input information from the pre-setting terminal 40. The pre-takeoff setting unit 310 acquires information input from each input unit of the input unit 410 of the pre-setting terminal 40 and sets each condition, etc. The pre-takeoff setting unit 310 includes a photography target competition setting unit 311, a photography target athlete setting unit 312, a course / geofence setting unit 313, a standby position setting unit 314, a takeoff position / landing position setting unit 315, a formation flight position setting unit 316, a leader / sub-aircraft setting unit 317, an evacuation setting unit 318, a priority setting unit 319, a setting error determination unit 310A, etc.

[0167] The filming target sport setting unit 311 sets the sport to be filmed in response to input information from the filming target sport information input unit 421 in FIG. 15 . The filming target athlete setting unit 312 sets the athlete to be filmed in response to input information from the filming target athlete information input unit 422. The course / geofence setting unit 313 sets the course and geofence in response to input information from the course / geofence input unit 423. The standby position setting unit 314 sets the standby position in response to input information from the standby position input unit 424. The takeoff position / landing position setting unit 315 sets the takeoff position and landing position in response to input information from the takeoff position / landing position input unit 425. The formation flight position setting unit 316 sets the formation flight position in response to input information from the formation flight position input unit 426. The leader / sub-aircraft setting unit 317 sets the leader aircraft and sub-aircraft in response to input information from the leader / sub-aircraft input unit 427. The save setting unit 318 sets a save action in response to input information in the save input unit 428. The priority setting unit 319 sets a priority in response to input information in the priority input unit 429.

[0168] The setting error determination unit 310A determines that a position setting error has occurred when the formation flight position, standby position, etc. input from the pre-setting terminal 40 is set to a position that deviates from the safety range set for each sport being photographed, and displays the details of the error on the error display unit 412 of the pre-setting terminal 40.

[0169] The aircraft status information acquisition unit 320 is a part that acquires the status of the aircraft 1 in the formation 10. The aircraft status information acquisition unit 320 has an aircraft flight status acquisition unit 321, a reception status acquisition unit 322, an obstacle information acquisition unit 323, etc. The aircraft flight status acquisition unit 321 acquires each piece of measurement information from the drone 1 by the flight status measurement unit 140 (Figure 11) of the drone 1. Examples of each piece of measurement information include position, direction, and speed. The obstacle information acquisition unit 323 acquires from the drone 1 obstacle detection information by the obstacle detection unit 130 of the drone 1. The obstacle detection information includes the competitor P, other aircraft, other obstacles, etc. An obstacle is any object that hinders the flight of the drone 1.

[0170] The reception status acquisition unit 322 acquires the reception status of the drone 1 control signal and the reception status of the positioning signal from the reception status detection unit 160 of the drone 1. The standby abnormality determination unit 330 determines that an abnormality has occurred when abnormality information is acquired from the drone 1 side. The standby abnormality determination unit 330 is used particularly for determining abnormalities in the standby state. The standby abnormality determination unit 330 acquires abnormality information from the flight function unit 120 or flight abnormality detection unit 124, camera abnormality detection unit 116, control communication abnormality detection unit 161, and positioning status abnormality detection unit 162 as functional units on the drone 1 side ( FIG. 11 ).

[0171] The competition information acquisition unit 340 acquires information on the competition / race, such as the start / end of the competition / race, from an external device.

[0172] The athlete information acquisition unit 350 acquires information about the athlete P. The athlete information acquisition unit 350 has a movement state acquisition unit 351 and a movement prediction unit 352. The movement state acquisition unit 351 acquires information such as the current position, speed, and movement direction of the athlete P from the athlete terminal 70. Instead of acquiring information from the athlete terminal 70, the movement state acquisition unit 351 may estimate the position, speed, movement direction, etc. of the athlete P from information such as the image captured by the drone 1, the position of the drone 1, and the orientation of the camera.

[0173] The control target command unit 360 has a command generation unit 361, a command execution unit 362, a command display unit 363, etc. The command generation unit 361 generates commands such as target values ​​for the positions (relative positions or absolute positions) of the leader drone and the sub drone of the formation 10, target values ​​for the speed, etc. The command execution unit 362 controls the operation of the drones 1 in the formation 10 in accordance with the generated commands. The command display unit 363 transmits command information to the control device 21 to display it.

[0174] The abnormality determination unit 370 determines various abnormalities, which will be described later. The abnormality determination unit 370 determines an abnormal state during tracking shooting mode. The abnormality determination unit 370 determines abnormalities in the leader device and the sub-device. A detailed flow of abnormality determination will be described later. The abnormality determination unit 370 includes a relative distance abnormality determination unit 371, a relative speed abnormality determination unit 372, a reception abnormality determination unit 373, a target value tracking abnormality determination unit 374, a geofence deviation determination unit 375, and the like.

[0175] The evacuation action determination unit 380 determines the evacuation action to be taken when an abnormality occurs in the leader aircraft of the formation 10, the evacuation action to be taken when an abnormality occurs in the sub aircraft, the evacuation action for each formation flight position, etc. A detailed flow of the evacuation determination will be described later.

[0176] The distribution video control unit 390 controls the distribution and display of the video captured by the drones 1 of the formation 10 (video data 250 in Figure 1) to the captured video display device 50.

[0177] [State Transition Diagram] Figure 19 is a state transition diagram of the present photography system. The present photography system has several modes for controlling the flight of the formation 10, etc. The present photography system switches between modes as appropriate. In this embodiment, the modes include a pre-takeoff preparation mode M1, a takeoff mode M2, a photography standby mode M3, a tracking photography mode M4, an emergency evacuation mode M5, and a landing mode M6.

[0178] Pre-takeoff preparation mode M1 is a preparation mode before the formation 10 takes off. The formation 10 takes off from a predetermined takeoff point. From pre-takeoff preparation mode M1, there is a transition to takeoff mode M2. Takeoff mode M2 ​​is a mode during which the formation 10 takes off. From takeoff mode M2, there is a transition to photography standby mode M3. Photography standby mode M3 is a state of preparation for photography before the start of a competition / race. From photography standby mode M3, there is a transition to tracking photography mode M4 after the start of a competition / race (transition s34). Furthermore, from photography standby mode M3, there is a transition to abnormality evacuation mode M5 in response to the detection of an abnormality (transition s35).

[0179] The tracking photography mode M4 is a mode in which the formation 10 is placed in a predetermined formation flight position (e.g., FIG. 2) and photographs the athlete P while flying and tracking the athlete P, who is the subject of photography. From the tracking photography mode M4, the mode transitions to the landing mode M6 upon the end of the competition or race (transition s46). Furthermore, from the tracking photography mode M4, the mode transitions to the emergency evacuation mode M5 in response to the detection of an abnormality (transition s45). Furthermore, from the tracking photography mode M4, the mode can also transition to the photography standby mode M3 (transition s43).

[0180] The landing mode M6 is a mode in which the aircraft lands at a predetermined landing point, and is the mode in which the formation 10 lands.

[0181] The emergency evacuation mode M5 is a mode in which a predetermined evacuation action is performed when an abnormality is detected in the tracking photography mode M4 or the photography standby mode M3. The emergency evacuation mode M5 transitions to the landing mode M6 (transition s56). The emergency evacuation mode M5 can also transition back to the tracking photography mode M4 (transition s54). The emergency evacuation mode M5 can also transition back to the photography standby mode M3 (transition s53).

[0182] A standard state transition is as follows. Before the start of a competition / race, the aircraft starts in pre-takeoff preparation mode M1, transitions to takeoff mode M2, and transitions from takeoff mode M2 ​​to photography standby mode M3. After the start of a competition / race, the aircraft transitions from photography standby mode M3 to tracking photography mode M4, and photographs are taken while following the competitor P during the competition / race. If there is no formation flight abnormality, the aircraft transitions from tracking photography mode M4 to landing mode M6 after the competition / race ends. If an abnormality is detected in photography standby mode M3, the aircraft transitions to abnormality evacuation mode M5 and then transitions to landing mode M6. If an abnormality is detected in tracking photography mode M4, the aircraft transitions to abnormality evacuation mode M5 and then transitions to landing mode M6. In other words, if a formation flight abnormality is detected, evacuation is performed for the aircraft 1 in which the abnormality was detected or the formation 10 including that aircraft, and then transitions to landing mode M6.

[0183] In a modified example, a photography standby mode M3 may be set even after the start of a competition or race. In this photography standby mode M3, the drone 1 flies following the athlete P, who is the subject of photography, at a certain distance, but does not photograph him / her. When photography is started from the photography standby mode M3, the drone 1 flies close to the athlete P, who is the subject of photography, and then transitions to the follow photography mode M4 (transition s34) and begins photographing the athlete P. Furthermore, when photography is temporarily stopped from that state, the drone 1 flies away from the athlete P and returns to the photography standby mode M3 (transition s43).

[0184] In a modified example, after transitioning from the tracking photography mode M4 to the abnormality evacuation mode M5 and performing evacuation, if the condition of the formation 10 or drone 1 improves and the formation flight abnormality is no longer present, it is possible to return to the tracking photography mode M4 (transition s54) and continue photographing. Similarly, after transitioning from the photography standby mode M3 to the abnormality evacuation mode M5 and performing evacuation, it is possible to return to the photography standby mode M3 (transition s53) if the condition of the formation 10 or drone 1 improves and the formation flight abnormality is no longer present.

[0185] [Flow: Pre-takeoff preparation mode] Figure 20 shows the operation flow of the pre-takeoff preparation mode M1 in this photography system. In step S101, the photography system sets the sport to be photographed. The sport to be photographed is selected from, for example, snowboarding, skiing, yachting, surfing, car racing, motorcycling, cycling, marathon, etc.

[0186] In step S102, the photography system sets the athlete to be photographed. The photography system sets identification information for the athlete P and identification information for the athlete terminal 70 including the sensor worn by the athlete P.

[0187] In step S103, the photography system sets a formation flight position in the follow photography mode M4. The formation flight position is, for example, as shown in Figures 2 to 4. At the same time, if the standby position at the start of the photography standby mode M3 is different from the formation flight position in the follow photography mode M4, the photography system also sets that standby position (e.g., Figure 21).

[0188] In step S104, the photography system sets a leader aircraft and a sub-aircraft of the formation 10. The photography system sets at least one leader aircraft and sets the other aircraft of the formation 10 as sub-aircraft. The leader aircraft is an aircraft that controls its relative position with respect to the competitor P, as described above. The sub-aircraft is an aircraft that controls its relative position with respect to the leader aircraft, as described above. Because the leader aircraft targets the relative position and relative distance with respect to the competitor P, it is preferable to set the aircraft in the formation 10 that flies closest to the competitor P as the leader aircraft.

[0189] In step S105, the filming system sets the competition course 61 and a geofence ( FIG. 24 ). The filming system sets the geofence to prevent the unmanned aerial vehicle 1 from flying outside a safe flight area. The filming system sets the geofence manually or automatically based on position information such as the terrain, immovable obstacles, and spectator positions.

[0190] In step S106, the photography system sets a landing position where the unmanned aerial vehicle 1 can safely land. The photography system sets the landing position manually or automatically based on position information such as the terrain, immovable obstacles, and spectator positions.

[0191] [Standby Position] Figure 21 is an explanatory diagram of the standby position of the formation 10 corresponding to the photography standby mode M3. Figure 21 shows a schematic perspective view of a first example of the standby position. Athlete P is at the starting point 62 of the course 61. In this example, the direction of travel immediately after the start is the Y direction (+Y). In this example, the standby positions of the drones 1 (1a, 1b, 1c, 1d, 1e) of the formation 10 are set at a predetermined distance ds1 behind (-Y) the starting point 62, where athlete P is located, in the Y direction; at the same position as athlete P in the X direction; and at a predetermined distance ds2 above (+Z) the position of athlete P (here, the road surface) in the Z direction.

[0192] In this example, the five drones 1 making up the formation 10 are arranged in series in the Z direction, which is the vertical direction. This is not limiting, and a standby position can be set. For example, the drones may be arranged in series in the horizontal direction. Alternatively, the drones may be arranged in a reduced distance between them while maintaining their relative positions, based on the formation flight position in the tracking photography mode shown in FIG. 2.

[0193] Figure 22 shows a second example of a standby position. In this second example, the standby position is established at a predetermined distance forward, in the direction of travel, from the position of athlete P at start point 62, and at a predetermined distance above that position. In this example, the standby position of the formation 10 is established at a predetermined distance ds3, for example, 10 m, forward (+Y) from the start point 62, and a predetermined distance ds4, for example, 10 m, vertically upward (+Z). This standby position is used as the reference position, and the formation flight positions of each drone 1 (1a to 1e) are established.

[0194] In the case of this standby position, a predetermined positional relationship has already been formed between the drones 1 at the time of standby, so that after the start of the competition, a formation flight position in tracking photography mode can be smoothly formed around the competitor P.

[0195] Figure 23 shows a third example of a standby position. In this third example, the standby position is established at a predetermined distance above the position of the contestant P at the starting point 62. In this example, the standby position of the formation 10 is established at a predetermined distance ds5, for example, 10 m, vertically above (+Z) the starting point 62. This standby position is used as the reference position to establish the formation flight positions of each drone 1 (1a-1e). In the case of this standby position, as in the case of Figure 22, the formation flight positions for the tracking photography mode can be smoothly formed around the contestant P after the start of the competition.

[0196] In this embodiment, the evacuation control is performed at least in the tracking shooting mode M4, but this is not limiting, and the evacuation control may also be applied to the shooting standby mode M3. For example, as shown in FIG. 21 , when multiple drones 1 are arranged in series in the shooting standby mode M3, an evacuation action may be performed if a state in which the distance between the drones 1 is not sufficiently secured is determined to be abnormal. In this case, a transition s35 from the shooting standby mode M3 to the abnormality evacuation mode M5 in FIG. 19 occurs.

[0197] [Geofence] FIG. 24 is an explanatory diagram of the setting of a geofence for the course 61. The geofence is established as a spatial region along the topography of the course 61, at a position covering the latitude, longitude, and altitude of the course 61. For example, the geofence may be set using the width and height of the course 61, which is represented by a line. The geofence is also set to avoid spaces that the drone 1 should not enter, such as forests, rocks, cliffs, and waterside areas. If a spectator area, a waiting area for personnel, or the like is installed near the course area, the geofence is set to avoid those areas. Furthermore, to prevent the drone 1 from flying near the height position of the athlete P, the geofence is set in the sky in the vertical direction so that the height distance from the road surface is equal to or greater than a predetermined height distance 2402.

[0198] [Example of Screen Display of Captured Video] Figure 25 shows an example of a screen display of captured video on the captured video display device 50. In most of the area of ​​the screen 2500 of the captured video display device 50, the video captured by the selected camera is displayed as a main camera video 2501. The main camera video 2501 corresponds to the main camera video display unit 511 (Figure 16) described above. In this example, the video captured by the selected camera is video captured by the camera Cc of the drone 1c located on the right side of the formation 10 in Figure 2, and is video with an angle of view capturing the right side of the athlete P on the course 61 in the direction of travel (video number 3, "Right Angle"). In this example video, the drone 1d located on the left is also visible in the background.

[0199] In one portion of the screen 2500, for example in the lower left, a sub-camera image 2502 is displayed, which corresponds to the other camera image display unit 512. In this example, the sub-camera image 2502 displays four camera images in a small area as images from the cameras of the drones 1 (1a, 1b, 1d, 1e) in the formation 10, excluding the image from the camera Cc of the drone 1c located on the right.

[0200] An enlarged view of sub-camera image 2502 is shown at the bottom of Figure 25. A user U2, such as a viewer, can select a desired sub-camera image from sub-camera image 2502 by operating the remote control of captured image display device 50, operating the cursor, inputting a number, touching, or the like. In response to this input operation, main camera switching unit 521 switches so that the selected sub-camera image is displayed in main camera image 2501. For example, when image No. 1, "Front Angle," is selected, the image from camera Ca of front drone 1a is displayed as main camera image 2501.

[0201] [Flow: Photography Standby Mode] Figure 26 shows the flow when in photography standby mode M3. When the photography system enters photography standby mode M3, it executes the flow in Figure 26. The main processing entity is the server 30, but it may also be each drone 1 in the formation 10.

[0202] In step S301, the photography system causes all drones 1 in the formation 10 to take off from a takeoff point (e.g., takeoff point 63 in FIG. 21 ) in takeoff mode M2 ​​and move them to a predetermined standby position. Examples of the predetermined standby position include the aforementioned example shown in FIG. 21 . Note that this standby position may be the same as the formation flight position in tracking photography mode M4, or may be a different position.

[0203] In step S302, the photography system checks the status of the drone 1 in the standby position and determines whether or not an abnormality has been detected in any of the drones 1. If no abnormality has been detected in any of the drones 1 (NO), the system proceeds to step S303, and if an abnormality has been detected in any of the drones 1 (YES), the system proceeds to step S307.

[0204] In step S303, the filming system notifies the control device 21 that all drones 1 in the formation 10 have completed standby. This notification allows the pilot U1 to recognize that all drones 1 in the formation 10 have completed standby. In step S304, the filming system checks whether the competition has started based on the acquired competition information, and if it has not started (NO), in step S305, the filming system causes all drones 1 to wait at their standby positions, and the process returns to step S302 and repeats the same process. If the competition has started (YES), in step S306, the filming system transitions all drones 1 in the formation 10 to tracking filming mode M4.

[0205] Meanwhile, in step S307, the filming system notifies the control device 21 that an abnormality has been detected in a drone 1. In step S308, the filming system checks whether the drone 1 in which the abnormality was detected is the leader drone. If the leader drone is abnormal (YES), in step S309, the filming system transitions all drones 1 in the formation 10 to abnormality evacuation mode M5. The control in step S309 above determines that all drones in the formation 10, including the sub-drones, are to be evacuated if the leader drone is abnormal, since the leader drone has the important role of determining its positional relationship with the competitor P.

[0206] If the drone 1 where the abnormality was detected is not the leader drone (NO), this corresponds to the drone 1 where the abnormality was detected being a sub-drone. If the sub-drone is abnormal, in step S310, the photography system transitions the abnormal sub-drone to landing mode M6 via abnormality evacuation mode M5 ( FIG. 19 ). The control in step S310 determines only the abnormal sub-drone as the evacuation target, because if a sub-drone is abnormal, even if some of the sub-drones that define themselves based on their position relative to the leader drone leave the formation 10, other drones can continue to follow and photograph them.

[0207] In step S311, the photography system checks whether the formation flight position of the abnormal sub-aircraft in step S310 is a predetermined position, for example, the front or rear position in FIG. 2. In this embodiment, this predetermined position is a position with a high priority among the formation flight positions. If this predetermined position is met (YES), in step S312, the photography system reassigns other drones 1 in other positions in the formation 10 (for example, left / right positions or positions directly above) to the formation flight position from which the abnormal sub-aircraft was evacuated. In other words, the formation flight position is changed or updated.

[0208] In the above control example, if the sub-aircraft determined to be abnormal is assigned to a position with a high priority within the formation flight position, the photography system reassigns a normal aircraft in another position to the higher priority position in order to ensure photography at the higher priority position. This allows photography to continue from the camera in the higher priority position. Details of the control using priority in step S311 will be described later.

[0209] In step S313, the photography system checks whether the competition has started based on the acquired competition information, and if it has not started (NO), in step S314, the photography system waits at the standby position, returns to step S302, and repeats the same process. If the competition has started (YES), in step S315, the photography system transitions the drones 1 in the formation 10 that are not malfunctioning, i.e., the drones remaining at the standby position, to tracking photography mode M4.

[0210] [Flow: Tracking Photography Mode] Figure 27 shows the flow of tracking photography mode M4. When the photography system enters tracking photography mode M4, it executes the flow of Figure 27. Note that the athlete P is moving with the start of the race.

[0211] In step S401, the photography system moves the drones 1 of the formation 10, which are in the standby position in photography standby mode M3, to a formation flight position (e.g., Figures 2 to 4) in tracking photography mode M4.

[0212] In step S402, the filming system acquires information and status of the athlete P, such as the ID, absolute position, speed, and direction of movement. For example, the drone 1 or the server 30 obtains this information by obtaining information from the athlete terminal 70 or the like. For example, the filming system obtains various pieces of information about the athlete P obtained from the status acquisition unit 710 of the athlete terminal 70 or an external device. There are at least three technical means for obtaining the athlete P's status, such as its position: the athlete terminal 70, which includes a sensor attached to the athlete P; acquisition from sensors such as the obstacle detection unit 130 of the drone 1; and estimation from images captured by the camera of the drone 1. Examples of obtaining the athlete P's position, etc., will be described later.

[0213] In step S403, the imaging system predicts the future position of the athlete P (for example, at the next time point) based on the information and state acquired in step S402.

[0214] In step S404, the photography system generates and updates control target values ​​related to the formation flight position and speed of the formation 10 for following the competitor P, based on the information and state predicted in step S403. The target values ​​generated in step S404 include position (at least one of relative distance, relative position, and absolute position) and speed. For the leader aircraft, a position where the relative distance or relative position from the competitor P satisfies a predetermined condition is generated as the target value (see, for example, FIG. 5). For the sub aircraft, a position where the relative distance or relative position from the leader aircraft satisfies a predetermined condition is generated as the target value.

[0215] In step S405, the photography system acquires the absolute positions of the leader drone 1 and the sub drone 1 in the formation 10. For example, the drone 1 or the server 30 determines the absolute positions by positioning or the like. For example, the absolute positions in the global coordinate system may be calculated based on relative position information using a pre-installed wireless base station 82 ( FIG. 1 ) as the reference point.

[0216] In step S406, the photography system acquires the relative position or distance of the leader camera to the position of the player P.

[0217] In step S407, the photography system acquires the relative position or relative distance of the sub-camera with respect to the position of the leader camera.

[0218] In step S408, based on the information acquired up to step S407, the imaging system generates and updates control commands for the leader drone 1 and the sub drones 1 of the formation 10. These control commands correspond to feedback control commands based on the difference between the target value and the current value.

[0219] In step S409, the photography system acquires information indicating the communication status of each drone 1 between the leader drone and the sub drone in the formation 10. For example, a certain drone 1 monitors the communication status with the other drones 1 and detects any abnormalities. The server 30 also monitors the communication status with each drone 1 through the control device 21 and detects any abnormalities. The photography system acquires information on the control communication status and positioning status acquired by the control communication abnormality detection unit 161 and positioning status abnormality detection unit 162 in FIG. 11 of the leader drone and the sub drone, for example.

[0220] In step S410, based on the information acquired in step S409, the photography system determines whether there is a formation flight abnormality with respect to each of the leader drone 1 and the sub drone 1 in the formation 10. Details will be described later.

[0221] Step S411 is a branch depending on the result of the determination in step S410. If it is determined that there is a formation flight abnormality (YES), the process proceeds to step S412. If there is no formation flight abnormality (NO), the process proceeds to step S413. In step S412, the photography system transitions to abnormality evacuation mode M5.

[0222] In step S413, it is confirmed whether the competition / race has ended, for example, whether competitor P has reached the finish line, and if it has not ended (NO), the process proceeds to step S414, and if it has ended (YES), the process proceeds to step S415. Note that, for confirmation of the start and end of the competition / race, for example, the competition information acquisition unit 340 (FIG. 18A) of the server 30 mentioned above acquires and grasps information on the status of the competition / race, such as competition start information and competition end information, and this information can be used.

[0223] In step S414, the tracking photography mode M4 continues, and in this case, the same process is repeated from step S42. In step S415, the photography system transitions to landing mode M6 or photography standby mode M3. For example, if the next competition by the next competitor continues, the photography system may transition to photography standby mode M3.

[0224] [Flow: Determining Formation Flight Anomalies in the Leader Aircraft] Figure 28 shows an example of the flow for determining formation flight anomalies in step S410 of Figure 27. Figure 28 shows an example of the process flow for determining formation flight anomalies in the leader aircraft. In the formation flight anomaly determination flow (Figures 28 and 29) of this embodiment, conditions with higher determination priority are determined first. The determination priority differs from the priority of the formation flight position. The determination priority has a first priority in considering the safety of the athlete P and determining whether to avoid collisions between the athlete P and the drone 1. The determination priority has a second priority in considering the safety of spectators and others around the course 61 and determining whether to prevent the drone 1 from departing from the geofence of the course 61. The determination priority has a third priority in considering the safety of the drone 1 and determining whether to avoid collisions between the drones 1. The example flow in Figure 28 includes five determination steps, steps S2801 to S2805, and the determination conditions associated with each step are illustrated as "Condition A1" to "Condition A5." Condition A1 has the highest priority for determination.

[0225] In step S2801, as condition A1, the photography system determines whether the communication status of the leader aircraft of the formation 10 is poorer than a predetermined level, such as the aforementioned communication abnormality. In step S2801, if there is an abnormality in the control communication of the leader aircraft, for example, if control communication is interrupted or delayed, it is determined to be a communication abnormality. Also, if an abnormality is detected in the positioning of the leader aircraft and competitor P, for example, if the positioning accuracy deteriorates, it is determined to be a positioning abnormality of the leader aircraft. Communication abnormalities will be described later. These correspond to the result of abnormality detection (YES) in the determination of condition A1 in step S2801. If there is a communication abnormality or the like in the leader aircraft (YES), the system proceeds to step S2806.

[0226] In step S2802, as condition A2, the photography system determines whether the relative position or relative distance of the leader aircraft with respect to competitor P is abnormal. Condition A2 is determined to be abnormal, for example, when the relative position or relative distance deviates from a predetermined range, in other words, the target range, in other words, when it is below a lower limit or exceeds an upper limit. Condition A2 is determined as follows: The photography system determines whether the state of the leader aircraft deviates from a control target value by a predetermined value or more. For example, if the difference between the leader aircraft's current position and the target position exceeds a predetermined value, an abnormality is determined. Alternatively, if the difference between the leader aircraft's current speed and the target speed exceeds a predetermined value, an abnormality is determined. The target position is defined as a relative position from the position of competitor P, or as an absolute position on the course 61. The target speed is defined as a speed relative to the speed of competitor P, or as an absolute speed on the course 61. The determination of condition A2 will be described later. If the relative position / relative distance is abnormal (YES), which satisfies condition A2, the process proceeds to step S2806.

[0227] Furthermore, the determination of condition A2 in step S2802 is not limited to the "athlete" set as the subject of photography, but may also be made for a group of athletes who are simultaneously traveling near the athlete. That is, the relative distance to be determined is the relative distance to any one of the athletes in the group, and if that relative distance falls below the lower limit of a predetermined range, it is determined to be abnormal. The determination of a group of athletes will be described later.

[0228] In step S2803, as condition A3, the photography system determines whether the current position of the leader camera is outside the range of the geofence. If it is outside, it is determined to be an abnormality, and the process proceeds to step S2806.

[0229] In step S2804, as condition A4, the photography system determines whether the relative speed of the leader camera with respect to competitor P is abnormal. For example, if the relative speed of the leader camera with respect to competitor P exceeds a predetermined value, it is determined to be abnormal. If the relative speed is abnormal (YES), the process proceeds to step S2806.

[0230] In step S2805, as condition A5, the photography system determines whether the relative position, relative distance, and relative speed of the leader drone relative to the positions and speeds of the surrounding sub-drones are within a predetermined range, in accordance with the additional function (FIG. 10). If they are outside the predetermined range, an abnormality is determined, and the process proceeds to step S2806. For example, if the relative distance of the leader drone to the sub-drones is less than a predetermined value, an abnormality is determined. For example, in FIG. 10, even if the relative distance of the leader drone to competitor P is within the target range, the relative distance between the sub-drones and the leader drone may be too close depending on the positions of the leader drone and the surrounding sub-drones.

[0231] If the determination result for any of the above conditions A1 to A5 indicates an abnormality, the photography system determines in step S2806 that the leader aircraft has a formation flight abnormality. If the determination results for all of the above conditions A1 to A5 indicate no abnormality, the photography system determines in step S2807 that the leader aircraft has no formation flight abnormality.

[0232] [Flow: Determining Formation Flight Anomaly of the Sub-Aircraft] Figure 29 shows an example of the flow for determining formation flight anomalies of the sub-aircraft, as an example of the flow for determining formation flight anomalies in step S410 in Figure 27. In step S2901, the image capture system determines that there is an anomaly if the communication state of the sub-aircraft is poor or if there is a communication anomaly or a positioning anomaly, as condition B1, and proceeds to step S2907.

[0233] In step S2902, the photography system determines, as condition B2, that the relative position or distance of the current position of the sub camera to competitor P is outside a predetermined range, and proceeds to step S2907. This determination includes a judgment on the group of competitors, as in the flow for the leader camera described above.

[0234] In step S2903, the imaging system determines that an abnormality has occurred if the current position of the sub-camera deviates from the geofence range as condition B3, and proceeds to step S2907.

[0235] In step S2904, the photography system determines that an abnormality has occurred if the relative distance or relative position of the sub-camera to the leader camera deviates from a predetermined range as condition B4, and proceeds to step S2907.

[0236] In step S2905, the photography system determines that an abnormality has occurred if the relative speed of the sub-camera with respect to the leader camera exceeds a predetermined value as condition B5, and proceeds to step S2907.

[0237] In step S2906, the photography system determines that there is an abnormality as condition B6, corresponding to the additional function (Figure 10), if the relative distance, relative position, or relative speed of the target sub-aircraft to other sub-aircraft in the formation 10 is outside a specified range, and proceeds to step S2907.

[0238] If the determination result for any of the above conditions B1 to B6 indicates an abnormality, the photography system determines in step S2907 that the sub-aircraft has a formation flight abnormality. If the determination results for all of the above conditions B1 to B6 indicate no abnormality, the photography system determines in step S2908 that the sub-aircraft has no formation flight abnormality.

[0239] [Formation flight abnormality] The various abnormalities described above are associated with formation flight abnormalities. A formation flight abnormality is an abnormality related to flight in a predetermined positional relationship with respect to the contestant P when flying the formation 10 in tracking photography mode M4. A formation flight abnormality corresponds to a state in which it is determined that there is a risk of contact or collision between the contestant P and the drone 1, or contact or collision between the drones 1. The following states correspond to formation flight abnormalities:

[0240] A state in which the relative position, distance, speed, etc. of the unmanned aerial vehicle 1 is outside a predetermined range (for example, a target range), or a state in which the difference between the absolute position or absolute speed of the unmanned aerial vehicle 1 and the target value exceeds a predetermined value. In other words, a state in which the position or speed of the unmanned aerial vehicle 1 does not satisfy a predetermined condition.

[0241] A state in which the communication of the unmanned aerial vehicle 1 is abnormal or malfunctioning. A state in which the positioning performance of the unmanned aerial vehicle 1 has deteriorated. With regard to the leader aircraft that acquires and understands positioning information from the athlete P's athlete terminal 70, if the positioning performance of the athlete terminal 70 has deteriorated, the corresponding leader aircraft is deemed to be abnormal.

[0242] [Understanding the Athlete's Position, etc.] Figure 30 is an explanatory diagram of technical means for determining the position, speed, direction, etc. of athlete P. In this example, some technical means are shown for the drone 1a in the forward position to determine the position, speed, and direction of athlete P. Athlete P has a position, speed, and direction P101. The athlete terminal 70 of athlete P obtains position information through GNSS positioning, and obtains information such as speed and direction through sensors.

[0243] First, the drone 1a can obtain the position, speed, and direction P101 of the athlete P by obtaining the ID, GNSS positioning information, and the like from the athlete terminal 70 based on wireless communication with the athlete terminal 70 of the athlete P. Also, as described above, a device (such as a transponder) installed in the competition environment 60 may measure the position of the athlete P, and the drone 1 and the server 30 may obtain information from that device via communication.

[0244] Furthermore, if the drone 1a is equipped with various sensors such as the aforementioned LiDAR, IR sensor, ToF sensor, and other ranging sensors, the position, speed, and direction P101 of the athlete P can be determined by measuring the athlete P using these various sensors.

[0245] In addition, the drone 1a can estimate and grasp the position, speed, and direction P101 of the athlete P based on the image of the athlete P captured by the camera Ca at a front angle of view and the control information of the camera Ca (e.g., orientation, zoom amount, etc.) using known image analysis processing, etc.

[0246] The drone 1a can perform tracking control of the formation flight position according to the position, speed, and direction P101 of the athlete P grasped by the above technical means. Specifically, the filming system can generate target ranges for the position, speed, and direction of the drone 1a (including the camera Ca) in a relative positional relationship with the athlete P as control target values.

[0247] In this embodiment, the leader device identifies and specifies the individual athlete P who is the subject of the image capture, and follows him / her. The athlete terminal 70 has an ID (athlete ID) that can identify the athlete P. The leader device may obtain the athlete ID, etc. from the athlete terminal 70. The leader device may also estimate the athlete ID by analyzing the content of the camera image.

[0248] [Communication Abnormality] Figure 31 is an explanatory diagram of an example of a communication abnormality. Here, communication abnormalities are broadly categorized into control communication abnormalities and positioning accuracy abnormalities. Control communication abnormalities include abnormalities in control communication with other unmanned aerial vehicles 1 and abnormalities in control communication with the athlete terminal 70. Positioning accuracy abnormalities include abnormalities in the positioning of the unmanned aerial vehicle 1 and abnormalities in the positioning of the athlete terminal 70. Control communication abnormalities include, for example, communication interruptions and communication delays. Positioning accuracy abnormalities include reduced positioning accuracy, reduced GNSS signal strength, the occurrence of multipath, etc.

[0249] In a formation 10 similar to that shown in FIG. 2 , drone 1a is the leader drone, and the other drones are sub-drones. For example, if the leader drone 1a experiences an abnormality in control communication with sub-drone 1c (e.g., communication from drone 1c is interrupted), it determines that drone 1c is abnormal. Furthermore, if the sub-drone 1d experiences an abnormality in control communication with leader drone 1a (e.g., communication from drone 1a is interrupted), it determines that drone 1a is abnormal. Furthermore, if the leader drone 1a's positioning accuracy is lower than a predetermined value, it determines that it is abnormal. Furthermore, if the leader drone 1a's positioning accuracy is lower than a predetermined value based on communication with the athlete terminal 70 (i.e., the athlete P associated with it), it determines that it is abnormal. Furthermore, if the sub-drone 1b's positioning accuracy is lower than a predetermined value, it determines that it is abnormal.

[0250] [Determining the relative distance to the competitor] Figure 32 is an explanatory diagram of determining the relative distance to the competitor P as a means of determining abnormalities in formation flight. In particular, an example related to step S2802 in Figure 28 and step S2902 in Figure 29 is shown. This example shows a case where the relative distance between the drone 1a in the front position and the competitor P is determined on an XY plane view. This example also corresponds to Figure 5.

[0251] The target control range for the position of the drone 1a in the forward position relative to the position PP1 and movement direction PD1 of the contestant P1 can be expressed as dpmin≦dp≦dpmax. dp is the relative distance between the position of the drone 1a and the position PP1 of the contestant P1. dpmin is the lower limit of the target range, and dpmax is the upper limit of the target range. While this depends on the competition, one example is 3m≦dp≦5m, 10m≦dp≦15m, etc.

[0252] The target range can be expressed as follows: If the target position is cp and the allowable positive and negative range around the target position is ±w, the target range can also be expressed as the range from cp-w to cp+w.

[0253] The aforementioned tracking control (e.g., FIG. 8B ) controls the position of the drone 1a so that it is within such a target range. If the drone 1a falls outside the target range, in other words, if it cannot maintain being within the target range, it is determined that the aforementioned formation flight abnormality has occurred.

[0254] The determination using the target range includes cases where the relative distance dp is outside the target range when it is less than the lower limit dp (dp<dp) or when it exceeds the upper limit dp (dp>dp). The determination of whether the relative distance dp is less than the lower limit dp is also referred to as a close determination. The determination of whether the relative distance dp is more than the upper limit dp is also referred to as a far determination.

[0255] In the embodiment, at least a proximity determination is performed, but it is more preferable to additionally perform a distance determination.

[0256] The same is true for determining the relative distance of the sub-drone to the leader drone. Furthermore, in addition to determining the relative distance, a determination using a target range can also be applied to the relative speed of drone 1 relative to the speed of contestant P. For example, if contestant P's speed is Vp and drone 1a's speed is Va, the target range for the relative speed is generated by setting the relative speed of drone 1a as (Va-Vp). If the relative speed (Va-Vp) falls outside the target range, it is determined that a formation flight abnormality has occurred.

[0257] [Determining Proximity to a Group of Athletes] A supplementary note will be provided regarding the determination of condition A2 in step S2802 of FIG. 28 and the determination of condition B2 in step S2902 of FIG. 29. The "athlete" considered in the above determination is not limited to a single specific athlete P set as the subject of filming. Depending on the competition and the situation, multiple athletes P may be present in close proximity at the same time. Multiple athletes P may be included as a group of athletes in the footage captured by the camera of drone 1. Such a group of multiple athletes P is a target for which the risk of collision should be considered. In this case, depending on the situation, it may be difficult for the leader drone to identify and track individual athletes P. In this case, the leader drone can simply follow the group of athletes, determine any abnormalities in the group of athletes, and control its evacuation to move away from the group of athletes, even if it is unable to identify the individual athletes. For example, in the judgment of condition A2 in step S2802, if the relative distance between the leader drone 1a and any of the athletes P in the group of athletes becomes smaller than the lower limit of the target range, it is judged to be an abnormality in proximity.

[0258] FIG. 33 shows an example of proximity determination with a group of athletes. In the example of FIG. 33, suppose athletes P are athletes P1, P2, P3, P4, and others running in a competition such as a marathon. The formation 10 initially targets athlete P1, but then other athletes P2 to P4 move in close to athlete P1. Drone 1a, the leader, captures the rear (-Y) direction with camera Ca. In addition to athlete P1, athletes P2 to P4 are also captured within the camera Ca's capture range. Here, athletes P1 to P4 are considered a group of athletes PG. The group of athletes PG is captured in the video captured by camera Ca.

[0259] The relative distance to athlete P is dp. For example, the relative distance to athlete P1 of drone 1a is dp1, and the relative distance to athlete P2 is dp2. The target range for the relative distance dp is dpmin≦dp≦dpmax, as in FIG. 32 . Normally, when the detected relative distance dp falls outside this target range, for example, when the relative distance dp1 is less than the lower limit dpmin as a proximity determination, it is determined that the drone is too close to athlete P, and an abnormality regarding the proximity has occurred.

[0260] Meanwhile, suppose that another competitor P2 in the group of competitors PG moves forward from competitor P1, and the relative distance dp2 becomes smaller than the relative distance dp1, as shown in the figure. In this case, drone 1a uses the closest relative distance dp2 to determine whether drone 1a is in proximity to the group of competitors PG. That is, if the relative distance dp2 is less than the lower limit dpmin, a proximity abnormality is also determined.

[0261] [Remote Determination] Furthermore, unlike the proximity determination described above, in which only a small relative distance is detected as an abnormality, an excessively large relative distance may also be detected as an abnormality (remote abnormality).

[0262] FIG. 34 shows an example of distance determination. In the example of FIG. 34, attention is focused on drone 1b, a sub-drone located at the rear. The relative distance to athlete P is dy. Assume that drone 1b is a distance dy1 behind (-Y) athlete P1, who is the subject of the image capture. Also, assume that another athlete P2 is traveling behind athlete P1. In this case, although drone 1b maintains a predetermined altitude, the relative distance dy2 to the other athlete P2 becomes small, increasing the risk of collision. Alternatively, if there are multiple formations 10, such as for each athlete P, the risk of collision with, for example, the leading drone of another similar formation 10 increases.

[0263] Therefore, when determining the relative distance of drone 1b, if the distance dy1 to competitor P1 is outside the target range, i.e., greater than the upper limit value dpmax (dy1>dpmax), it is determined to be too far away and an abnormality (remote abnormality) is detected.

[0264] As for the control, both the proximity determination to the group of athletes and the remote determination as described above may be applied. Remote determination can also be applied to the group of athletes.

[0265] [Evacuation Mode in Case of Abnormality] Figure 35 shows the control flow for the emergency evacuation mode M5. In step S501, the photography system checks whether the aircraft experiencing the formation flight abnormality is the leader aircraft or the sub-aircraft. If it is the leader aircraft, the process proceeds to step S502, and if it is the sub-aircraft, the process proceeds to step S507.

[0266] In step S502, the photography system decides to evacuate all aircraft in the formation 10, including the leader and sub aircraft, and notifies the control device 21 of the evacuation.

[0267] In step S503, the photography system increases the altitude of all drones 1, including the leader and sub drones, of the formation 10 to be evacuated to a predetermined altitude as an example of an evacuation action. Alternatively, as an example of an evacuation action, the photography system increases the altitude from the pre-evacuation altitude to the post-evacuation altitude by a predetermined altitude difference or more.

[0268] In step S504, the photography system checks whether the relative distance of the leader aircraft from competitor P is greater than a predetermined value. In other words, this is a check to see whether sufficient evacuation has been performed as an evacuation action. If not (NO), the evacuation is insufficient, so in step S505, the photography system, for example, makes the aircraft wait in the air. Then, the process returns to step S504. If the leader aircraft is far away (YES), the evacuation is sufficient, so in step S506, the photography system transitions the aircraft to landing mode M6.

[0269] In step S507, the photography system decides to evacuate only the sub-machine with an abnormality, and notifies the control device 21 of this evacuation.

[0270] In step S508, the photography system increases the altitude of the sub-camera to be evacuated to a predetermined altitude as an example of an evacuation action.

[0271] In step S509, the photography system checks whether the relative distance of the sub-aircraft to the leader aircraft is greater than a predetermined value. In other words, this is a check to see whether sufficient evacuation has been performed as an evacuation action. If the distance is not sufficient (NO), the evacuation is insufficient, so in step S510 the photography system makes the aircraft wait in the air and then returns to step S509. If the distance is sufficient (YES), the evacuation is sufficient, so in step S511 the photography system transitions the aircraft to landing mode M6. This control lands the sub-aircraft after the relative distance from the leader aircraft, which continues to follow competitor P, becomes sufficiently large.

[0272] The evacuation actions in steps S503 and S508 are basically evacuation actions in which the drone 1 is moved in a direction away from the contestant P, and in the specific example in this embodiment, this is an altitude increase operation. The details of the evacuation action can be defined in various ways. The altitude increase operation as the evacuation action in steps S503 and S508 may be performed by either automatic control or manual operation by the pilot U1.

[0273] In this embodiment, if only a sub-drone in the formation 10 has an abnormality, only the abnormal sub-drone is evacuated in step S507, but this is not limited to this. All drones, including the abnormal sub-drone and the leader drone, may be evacuated. In a mode in which only the abnormal sub-drone is evacuated, the remaining drones 1 can continue to take pictures without evacuating. In a mode in which all drones are evacuated, emphasis can be placed on safety. In another modified example, all sub-drones may be evacuated, leaving only some of the drones 1, for example the leader drone.

[0274] [Example of Evacuation Action (1)] Figure 36 is an explanatory diagram of an example of an evacuation action in abnormality evacuation mode M5, and is a Y-Z plane view seen from the right side relative to the movement direction PD1 of athlete P. First, state (1) is a state in which, in tracking photography mode M4, the formation 10 is moving and photographing athlete P while following him / her in a formation flight position configuration such as those shown in Figures 2 to 4 above. In this state, if an abnormality is detected in the leader drone 1a, for example, it is decided to evacuate the entire formation 10 according to Figure 35. If an abnormality is detected in the sub-drone, it is decided to evacuate only that sub-drone. This example shows a case in which the entire formation 10 is evacuated.

[0275] State (2) shows a state in which the entire formation 10 has retreated from state (1) by moving upward (+Z) in an increasing altitude. For example, each drone 1 in the formation 10 has moved upward a distance DH1 from its position in (1). Note that "upward" here refers to the vertical direction (+Z direction), but this is not limited to this and may also refer to a direction perpendicular to the road surface of the course 61. During this retreat movement, the contestant P is traveling along the course 61 in a moving direction PD1. The relative distance dap is the relative distance between the leader drone 1a and the contestant P at the time of retreat. Note that this retreat, which is an increasing altitude movement, is an action in which the drone stops following the contestant P (described below).

[0276] The filming system transitions the formation 10 to landing mode M6 after the relative distance dap between the leader drone 1a and the contestant P during evacuation has become sufficiently large. State (3) shows a state in which, from state (2), the formation 10 transitions to landing mode M6 after the relative distance dap has reached a predetermined value or greater, and the formation 10 lands at the set landing point 64. In Figure 36, an example of the formation's evacuation operation is shown in which the formation moves upward in altitude away from the contestant who is the subject of filming, and then lands after moving sufficiently away from the contestant. However, the formation's evacuation operation is not limited to this, and any flight that at least increases the relative distance between the contestant and the formation is sufficient, and does not necessarily have to involve a landing operation.

[0277] [Example of Evacuation Action (2)] Figure 37 shows another example of an evacuation action, in which only one sub-drone in the formation 10 that is experiencing a formation flight abnormality is evacuated. State (1) shows a state in which the drone is following the contestant P in tracking and photography mode M4. In this example, assume that the sub-drone 1c located on the right has an abnormality. In this case, state (2) shows a case in which only drone 1c is evacuated upward (+Z) by a distance DH2 in abnormality evacuation mode M5. This evacuation, which involves an upward movement in altitude, is an action in which the drone stops following contestant P. The other drones in the formation 10 continue to follow contestant P in tracking and photography mode M4. The relative distance dcp is the relative distance between the evacuated drone 1c and contestant P. State (3) shows a state in which drone 1c lands at landing point 64 in landing mode M6 after the relative distance dcp has become sufficiently large.

[0278] [Types of Evacuation Actions] Figures 38A and 38B are explanatory diagrams showing types of evacuation actions. Two types of evacuation actions are shown, depending on whether the player stops following the player P or continues following the player P during the evacuation action. In other words, the difference between these two types of evacuation actions is whether they are defined as absolute movements independent of the player P's movements, or as relative movements following the player P's movements. Either type of evacuation action may be applied.

[0279] Figure 38A shows an example of an evacuation action to stop following the contestant P in a Y-Z plane view. In this example, if the sub-drone 1c located on the right develops an abnormality, only that drone 1c is evacuated. In the abnormality evacuation mode M5, the drone 1c first performs movement 3801, ascending upward (+Z) to a predetermined altitude. During movement 3801, the drone 1c stops following the contestant P and simply moves vertically (+Z) in space. During movement 3801, the contestant P and the other drones 1 in the formation 10 are traveling in the movement direction PD1 (+Y). Therefore, the distance between the ascending drone 1c and the contestant P and the other drones 1 gradually increases over time. The drone 1c may hover at an upper position Lc1. The distance dcp is the distance between the ascending drone 1c and the contestant P.

[0280] Furthermore, drone 1c at upper position Lc1 may perform movement 3802, such as backward (-Y), which is opposite to movement direction PD1. Movement 3801 and movement 3802 are both movements away from contestant P and have a low risk of contact with other drones 1 in the formation 10. The evacuation movement path of drone 1c is selected so that there are no other drones 1 on it, or so that a sufficiently large relative distance is maintained between drone 1c and the other drones 1. As shown in FIG. 38A , the evacuation action may be a movement in which drone 1c stops following contestant P.

[0281] Figure 38B shows an example of an evacuation action that maintains tracking of the athlete P. In this example, if the sub-drone 1c on the right experiences an abnormality, only that drone 1c is evacuated. In the abnormality evacuation mode M5, the drone 1c first performs upward movement 3803 away from the athlete P while maintaining its relative relationship to the athlete P, i.e., its positional relationship diagonally upward to the right. The movement 3803 maintains tracking of the athlete P, i.e., includes a component of movement in the movement direction PD1 (+Y), and moves diagonally upward (+Y, +Z) in space. During this movement 3803, the athlete P and the other drones 1 in the formation 10 are traveling in the movement direction PD1. Therefore, the positional relationship between the ascending drone 1c, the athlete P, and the other drones 1 is maintained in the X and Y directions, but the distance increases only in the Z direction. The distance dcp is the distance between the ascending drone 1c and the athlete P. Drone 1c may continue to move in the +Y direction to maintain tracking so as to remain at an upper position Lc2 relative to the moving athlete P.

[0282] Furthermore, drone 1c at upper position Lc2 may perform movement 3804, such as backward (-Y), which is opposite to the movement direction PD1. Movement 3804 may be performed by reducing its speed in the forward / backward direction (Y direction) or setting it to zero (hovering), thereby moving relatively behind contestant P. Movement 3803 and movement 3804 are both movements away from contestant P and pose a low risk of contact with other drones 1 in the formation 10. The evacuation movement path for drone 1c is selected so that there are no other drones 1 on the path or so that a sufficient relative distance from the other drones 1 is maintained. As shown in FIG. 38B, the evacuation action may be performed while maintaining follow-up to contestant P.

[0283] In addition, in any evacuation action, when some of the drones 1 in the formation 10 are evacuated, an appropriate evacuation route, etc. is controlled so that collisions do not occur between the evacuating drones and the drones 1 that do not evacuate.

[0284] [Evacuation Actions for Each Formation Flight Position] The above-mentioned evacuation actions for each formation flight position can be set in advance using the formation flight position input unit 426 ( FIG. 15 ) and the formation flight position setting unit 316 ( FIG. 18A ). An example of an evacuation action for each formation flight position for the formation 10 shown in FIG. 2 is shown below.

[0285] The example in Figure 39 shows an example of an evacuation action for the drone 1a in the front position. In the example in Figure 39, the drone 1a in the front position performs movement 3901, increasing its altitude and ascending upward (+Z). After movement 3901, the drone 1a may also move left or right in the left-right direction (X direction). From that left or right position, the drone may also move further backward (-Y). In another example, after the upward movement 3901, the drone may move backward (-Y) 3902. However, an appropriate altitude must be selected to avoid collision with the drone 1e directly above. In another example, the drone 1a in the front position may move left or right away from the contestant P without increasing its altitude, or may then move backward. Figure 39 illustrates these evacuation examples together. Each evacuation movement route is designed to reduce the risk of contact with other drones 1.

[0286] The example in Figure 40 shows an example of the evacuation action of drones 1c and 1d located on the left and right. As an evacuation action, drones 1c and 1d first ascend upward (+Z) 3903. After that, drones 1c and 1d may move backward (-Y). After movement 3903, drone 1c may move to the right, away from contestant P and the formation 10, or drone 1d may move to the left, away from contestant P and the formation 10. After that, drones 1c and 1d may move backward (-Y). In another example, drones 1c and 1d located on the left and right may move left or right, away from contestant P, without increasing their altitude. Furthermore, after that left or right movement, they may move backward. These examples are illustrated together in Figure 40.

[0287] The example in Figure 41 shows an example of the evacuation action of drone 1b in the rear position. As the evacuation action of drone 1b, first, movement 3904 is performed in which drone 1b increases its altitude and ascends upward (+Z). After that, drone 1b may move backward (-Y). After movement 3901, drone 1b may move to the right or left. In another example, drone 1b may move backward (-Y) or to the right or left without increasing its altitude. Figure 41 illustrates these examples together.

[0288] The example in Figure 42 shows an example of the evacuation action of drone 1e located directly above. As the evacuation action of drone 1e, first, it may be movement 3905, in which it increases its altitude and ascends. Alternatively, if the altitude is sufficiently high from the start, it may be movement 3906 backward (-Y) without ascending. The evacuation action may also be movement to the left or right, away from player P. Figure 42 illustrates these examples together.

[0289] The examples of the retreat actions in Figures 39 to 42 are valid for either the type shown in Figure 38A or Figure 38B. For example, the retreat shown as a backward (-Y) movement in Figure 40 corresponds to movement 3802 in Figure 38A and movement 3804 in Figure 38B. Such retreat movement may be an absolute movement in which the drone stops following the athlete P, as in Figure 38A, or a relative movement in which the drone maintains following the athlete P. For example, for a drone 1 that moves forward (+Y) at a predetermined speed while following a athlete P moving forward (+Y), during retreat, the drone 1's speed in the forward / backward direction (Y direction) can be reduced or set to zero, resulting in a backward (-Y) movement relative to the athlete P.

[0290] [Landing Mode] Figure 43 shows the control flow for landing mode M6. In step S601, the photography system accepts input of a landing action from the control device 21. For example, the server 30 or the drone 1 transmits an instruction or notification to the control device 21 to transition to landing mode M6 based on the completion of the aforementioned evacuation action. The control device 21 follows the instruction and waits for input of a landing action (in other words, input of a landing control instruction) from the pilot U1. The control device 21 allows the pilot U1 to specify landing actions such as landing permission, hovering standby, soft landing on the spot, and propeller stop through input operations.

[0291] In step S602, the filming system checks whether a landing action has been input, and if so (YES), in step S603 the filming system executes a landing action corresponding to the input. The filming system lands the formation 10 at the landing site 64 in landing mode M6, for example, as shown in (3) of Figure 36. If no landing action has been input (NO), in step S604 the filming system causes the formation 10 to wait by hovering in the state after the evacuation action ((2) of Figure 36) for a predetermined period of time. Then, in step S605, after the predetermined period has elapsed, the filming system automatically switches the formation 10 to landing mode M6 and lands it at the landing site 64, even if no input has been received.

[0292] [Determining an Evacuation Target] Figure 44 is an explanatory diagram showing an example of determining an evacuation target based on an abnormal state in step S502 or step S507 of the flow in Figure 35. In particular, it shows a specific example of which aircraft in the formation 10 is determined to be abnormal and the evacuation target is determined. In the formation 10 similar to that in Figure 2, drone 1a is assumed to be the leader aircraft and the other aircraft are assumed to be sub-aircraft.

[0293] Example 1: First, the leader drone 1a performs an abnormality determination based on predetermined conditions. For example, if the leader drone 1a determines that it is abnormal, it determines that the entire formation 10, including the leader drone 1a, is to be subject to evacuation.

[0294] Example 2: Furthermore, if the leader drone 1a determines that another drone, for example drone 1c, is abnormal as a result of the abnormality determination, it determines at least drone 1c as the drone to be evacuated.

[0295] Example 3: The sub-drone 1c determines whether there is an abnormality under a predetermined condition. For example, if the sub-drone 1c determines that there is an abnormality, it determines that at least it is a target for evacuation.

[0296] Example 4: The sub drone 1d also performs an abnormality determination under predetermined conditions. For example, if the drone 1d determines that the leader drone 1a is abnormal, it determines that the entire formation 10, including the drone 1a, is to be targeted for evacuation.

[0297] [Determining Evacuation Targets: Modification] Figure 45 shows a modification example of a judgment between sub-aircraft when determining evacuation targets. If a certain sub-aircraft is determined to have a formation flight abnormality and there is a risk of collision with other sub-aircraft flying nearby, the other sub-aircraft may also be determined to be evacuation targets. In the example of Figure 45, first, assume that the sub-aircraft, drone 1c located on the right, is determined to have a formation flight abnormality when it leaves the target range and is determined to be an evacuation target. Furthermore, the filming system checks the relative distance between the abnormal drone 1c and other nearby drones 1. For example, drone 1c checks the distance DCB between drone 1c and drone 1b located behind it. Alternatively, drone 1b located behind it checks the distance DCB between drone 1c and drone 1c. If the distance DCB is less than a predetermined value, it is determined that there is a risk of collision. As a result, the filming system may also select drone 1b as an evacuation target, just to be safe.

[0298] [Priority] Figure 46 shows an example of the priority of formation flight positions and the corresponding camera positions and shooting angles of view. In a formation 10 similar to that shown in Figure 2, drone 1a at forward position La obtains video 4601 of competitor P captured at a front angle of view using rear-facing camera Ca. Drone 1b at rear position Lb obtains video 4602 of competitor P captured at a rear angle of view using forward-facing camera Cb. Drone 1c at right position Lc obtains video 4603 of competitor P captured at a right side angle of view using left-facing camera Cc. Drone 1d at left position Ld obtains video 4604 of competitor P captured at a left side angle of view using right-facing camera Cd. Although not shown, drone 1e (Figure 2) at directly overhead position Le obtains video of competitor P captured at a bird's-eye view angle using downward-facing camera Ce.

[0299] The image capture system can set a priority for each formation flight position using the priority setting unit 319 ( FIG. 18A ) or the like. Priority, in other words, is importance. In the example of FIG. 46 , a first priority, p1, is set for the image 4601 at the forward position La and front angle of view, a second priority, p2, is set for the image 4602 at the rear position Lb and rear angle of view, a third priority, p3, is set for the image 4603 at the right position Lc and right side angle of view, and a fourth priority, p4, is set for the image 4604 at the left position Ld and left side angle of view. A fifth priority, p5, is set for the image directly above position Le and the bird's-eye view. Priority p1 is the highest.

[0300] The priority can be freely set depending on the competition, taking into consideration the importance of the image of the angle of view. The angle of view here corresponds to the orientation, position, size, etc. of the competitor P. A priority can be set for the formation flight position that can capture an image of a predetermined angle of view. Note that the priority can be set independently of the settings for the leader and sub-vehicles.

[0301] In this example, the priority is set to each position without overlapping, but the same priority may be set to multiple positions. Also, in this example, the priority is set to all positions, but the priority may be set to only some positions.

[0302] [Control Using Priority] Figure 47 shows an example of reassigning formation flight positions during evacuation, as an example of control using priority as in Figure 46. In other words, it shows a control example in which an alternative drone 1 is deployed at the evacuation position. This corresponds to a specific example of the control of step S311 in Figure 26. Figure 26 shows an example of control in the photography standby mode M3, but control using priority can also be applied in the tracking photography mode M4.

[0303] In Figure 47, priorities such as those in Figure 46 are set for five drones 1 similar to those in Figure 2. The photography system may also set the need to continue photography, etc., for each priority. For example, a high priority of p2 or higher is set as a priority at which photography should be continued as much as possible. A low priority of p3 or lower is set as a priority at which photography does not necessarily need to be continued. In other words, if a formation flight abnormality occurs at a certain position and the aircraft at that position is evacuated, if the priority of that position is p2 or p1, a replacement aircraft from another position can be reassigned to continue photography at the high priority position.

[0304] In the example of Figure 47, first, in state (1), it is determined that drone 1b at rear position Lb with priority p2 is experiencing a formation flight abnormality. Drone 1b, the sub-drone, is selected as the target for evacuation. The evacuated drone 1b then lands in landing mode M6.

[0305] Next, in state (2), an example is shown in which a drone 1 in another position is reassigned to the rear position Lb with priority p2 from which drone 1b has retreated. Since the rear position Lb with priority p2 is now vacant, the imaging system reassigns one sub-machine selected from other normal sub-machines with priority p3 or lower to the rear position Lb. In this example, the candidate sub-machines with priority p3 or lower are drone 1c in the right position Lc, drone 1d in the left position Ld, and drone 1e in the directly above position Le. For example, the imaging system selects drone 1e in the directly above position Le with priority p5, which has the lowest priority, and assigns the selected drone 1e to the rear position Lb. That is, the imaging system moves drone 1e from the directly above position Le to the rear position Lb. Then, in tracking imaging mode M4, the imaging system causes drone 1e's camera Ce to capture video facing forward with a rear-facing angle of view.

[0306] In the above control example, drone 1a, the leader drone with priority p1, is important and is therefore not allowed to change its position mid-flight. In this way, some drones may have fixed formation flight positions and others may have variable positions.

[0307] Another example of control related to the above-mentioned evacuation and substitution is as follows. In FIG. 47 , assume that the leader drone 1a with priority p1 is determined to have a formation flight abnormality. While the above-described control evacuates the entire formation 10, in this example, only the drone 1a in the front position is evacuated, and a replacement drone 1 is deployed in the front position. The filming system selects one normal sub-drone from the other sub-drones 1 with priority p2 or lower and deploys it at the front position La. For example, drone 1e with priority p5 is selected and deployed at the front position La. This allows drone 1e's camera Ce to continue capturing video at a front angle of view from the front position La. In other words, the viewer can continue to be provided with video captured at a front angle of view.

[0308] Furthermore, when the leader drone is evacuated as described above and another sub-drone is reassigned to the evacuated location as a replacement, a role change is performed to change the sub-drone to a new leader drone. Because at least one drone must be a leader drone in the formation 10, if the only leader drone is evacuated, a new leader drone must be set in its place. In the embodiment, it is assumed that the multiple drones 1 in the formation 10 have similar or equivalent performance. Therefore, changing a sub-drone to a new leader drone does not pose a performance problem. Furthermore, if there is only one leader drone in the formation 10 or if a machine with specific performance is used as the leader drone, the above-described leader drone evacuation and replacement may not be permitted. If there are two or more leader drones in the formation 10, the above-described leader drone evacuation and replacement may be permitted.

[0309] [Modification: Control of Temporary Evacuation and Return] As a modification, the following is also possible. The evacuation action from the emergency evacuation mode M5 described above basically involves transitioning to the landing mode M6 (transition s56 in FIG. 19 ), but this is not limited to this. As a modification, control may be performed to return from the emergency evacuation mode M5 to the tracking photography mode M4 (transition s54 in FIG. 19 ).

[0310] 48 shows an example of control of temporary evacuation and return in a modified example. First, in the case of state (1), when, for example, drone 1c in right position Lc1 of the formation 10 is experiencing a formation flight abnormality, it is determined to be a target for temporary evacuation. In this case, the abnormality may be, for example, a minor abnormality such as a communication malfunction, or a state in which the degree of abnormality is not very high.

[0311] (2) indicates an escape action. As an escape action, the filming system moves drone 1c away from athlete P and the formation 10, for example, to position Lc2 on the outer right (+X). Position Lc2 is farther from athlete P than position Lc1. This temporary escape maintains tracking of athlete P (similar to FIG. 38B) and includes a movement component in the movement direction PD1 (+Y). After the movement, position Lc2 follows athlete P and the formation 10 while maintaining a certain relative distance, and filming has stopped. The filming system monitors the status of drone 1c at position Lc2.

[0312] (3) shows a case where, if the communication problem or other condition of drone 1c improves, drone 1c is returned to its original position Lc1 in tracking photography mode M4. The photography system determines that drone 1c is in a state where there are no formation flight abnormalities and moves it from position Lc2 to position Lc1. Position Lc1 after the return is a position where drone 1c follows competitor P. Drone 1c can continue photographing competitor P from position Lc1 after the return.

[0313] In the above control example, the temporary evacuation position is the outer right position Lc2, but this is not limited thereto, and as with the above-described example of the evacuation action, the temporary evacuation position can be set in various positional relationships depending on the formation flight position. For example, the temporary evacuation position may be a position at a higher altitude. Furthermore, although the above control example is for temporary evacuation of only one drone 1, temporary evacuation of multiple drones 1 is also possible.

[0314] [Modification: Evacuation and Camera Zoom Control] Figure 49 shows a modification example of control that combines an evacuation action and camera zoom control. For example, in tracking shooting mode M4, the sub-drone 1c at the right position Lc1 is photographing the athlete P from the right side with a camera Cc facing left. Here, for example, if it is determined that the drone 1c is experiencing a formation flight abnormality, the shooting system will evacuate the drone 1c. For example, the evacuation action of the drone 1c is movement 4900 to the outside right (+X), away from the athlete P. Figure 49 shows a schematic diagram of an image 4901 taken by camera Cc at position Lc1 before evacuation, a schematic diagram of an image 4902 taken by camera Cc at position Lc2 after evacuation, with the right side angle of view, when no special control is applied, and a schematic diagram of an image 4903 taken by camera Cc at position Lc2 after evacuation, with the right side angle of view, when additional camera zoom control is applied.

[0315] In normal tracking shooting mode M4, the orientation and zoom amount of camera Cc are controlled so that the position and size of contestant P in the camera image are optimal, as in captured image 4901. After evacuation, if there is no particular control, the orientation and distance from camera Cc to contestant P change, so the position and size of contestant P in the image are no longer optimal, as in captured image 4902. After evacuation, the distance between camera Cc and contestant P increases, so the contestant P in the image becomes smaller. Furthermore, when drone 1c stops tracking, contestant P is captured in a position that is off-center in the image.

[0316] In contrast, in this modified example, the filming system additionally applies camera zoom control during evacuation. The filming system adjusts the orientation and zoom amount of the camera Cc during the evacuation movement 4900 so that the position and size of the athlete P captured in the image of the camera Cc are as optimal as possible. The filming system adjusts the orientation and zoom amount of the camera Cc in accordance with the evacuation action of the drone 1c, i.e., in accordance with the evacuation direction, distance, and speed. This allows for the provision of optimal filmed footage 4903, such as the filmed footage 4903, in which the athlete P is captured at an optimal size near the center of the filmed image 4903, both during and after evacuation. As described above, filming can be continued if possible while the drone 1 is being evacuated. Even if the same image as the original position cannot be obtained from the evacuation position, it is useful to be able to obtain image content that is as close as possible.

[0317] [Effects, etc.] As described above, the filming system and filming method of the first embodiment can provide a technology that ensures safety and improves the quality of the filmed video when multiple unmanned aerial vehicles 1 are used to track and film a competitor P. According to the first embodiment, the formation flight control described above can reduce the risk of contact or collision between the unmanned aerial vehicles 1 and the competitor P or nearby spectators, etc. According to the first embodiment, for example, even if the unmanned aerial vehicle 1 becomes abnormal, it can more reliably avoid contact or collision with the competitor P. Furthermore, according to the first embodiment, it is possible to ensure safety while improving the quality of the video filmed by the unmanned aerial vehicle 1. For example, it is possible to stabilize the distance of the unmanned aerial vehicle 1 from the competitor P and obtain video with a preferable filming angle, etc., so that the movements of the competitor P can be clearly understood.

[0318] Although the embodiments of the present disclosure have been specifically described above, they are not limited to the above-described embodiments and various modifications are possible without departing from the spirit of the present disclosure. In each embodiment, components can be added, deleted, or replaced, except for essential components. Unless otherwise specified, each component may be singular or plural. A combination of each embodiment and its modified examples is also possible. The program of the embodiment is a program for causing a computer (e.g., the server 30, the drone 1, the pilot device 20, or other equipment) to execute processing by the imaging system of the embodiment. The program of the embodiment may be provided stored in a storage medium / recording medium.

[0319] 1, 1a, 1b, 1c, 1d, 1e...unmanned aerial vehicle (drone), 10...formation, 20...pilot device, 21...pilot device, 22...monitor device, 30...server, 40...pre-setting terminal, 50...captured image display device, 60...competition environment, 61...course, 64...landing point, 70...athlete terminal, 90...communication network, 100...computer system, P...athlete, U1...pilot, U2...user, PD1...direction of movement, DH1...distance, dap...distance.

Claims

1. A photography system using unmanned aerial vehicles, comprising: a plurality of unmanned aerial vehicles that fly in pursuit of a subject to be photographed; each unmanned aerial vehicle in the plurality of unmanned aerial vehicles photographs the subject to be photographed with a camera to provide photographed footage; the plurality of unmanned aerial vehicles fly in a formation flight position in a predetermined positional relationship with the subject to be photographed; a first unmanned aerial vehicle among the plurality of unmanned aerial vehicles is controlled to control the flight of the first unmanned aerial vehicle so that at least one of the parameter values ​​of the relative position, relative distance, absolute position, relative velocity, and absolute velocity with respect to the subject to be photographed satisfies a predetermined condition; a unmanned aerial vehicle that does not satisfy the predetermined condition is determined to be abnormal in formation flight; and an evacuation action is taken to move the unmanned aerial vehicle that is abnormal in formation flight away from the subject to be photographed.

2. The imaging system according to claim 1, wherein the evacuation action includes at least movement in a direction increasing the altitude.

3. In the photography system described in claim 1, the first unmanned aerial vehicle acquires position information of the subject to be photographed and controls the position of the first unmanned aerial vehicle so that the relative position or relative distance of the first unmanned aerial vehicle to the position of the subject to be photographed satisfies a target value or target range.

4. The photography system according to claim 3, wherein all of the plurality of unmanned aerial vehicles are the first unmanned aerial vehicle.

5. An imaging system as described in claim 1, wherein the first unmanned aerial vehicle acquires information on the position and movement direction of the subject to be imaged, and controls the position of the first unmanned aerial vehicle so that the relative position or relative distance of the first unmanned aerial vehicle to the position and movement direction of the subject to be imaged satisfies a target value or target range.

6. An imaging system as described in claim 1, wherein the first unmanned aerial vehicle is judged to be in a formation flight abnormality when the relative position or relative distance of the first unmanned aerial vehicle to the subject to be photographed does not satisfy a target value or target range, or when the difference between the absolute position of the first unmanned aerial vehicle and a target absolute position exceeds a predetermined value.

7. An imaging system as described in claim 1, wherein the first unmanned aerial vehicle is judged to be in a formation flight abnormality when the relative speed of the first unmanned aerial vehicle with respect to the subject to be photographed does not meet a target value or target range, or when the difference between the absolute speed of the first unmanned aerial vehicle and a target absolute speed exceeds a predetermined value.

8. An imaging system as described in claim 1, wherein a second unmanned aerial vehicle among the plurality of unmanned aerial vehicles, which is not the first unmanned aerial vehicle, acquires position information of the first unmanned aerial vehicle and controls the position of the second unmanned aerial vehicle so that the relative position or relative distance of the second unmanned aerial vehicle to the position of the first unmanned aerial vehicle satisfies a target value or target range.

9. An imaging system as described in claim 1, wherein a second unmanned aerial vehicle among the plurality of unmanned aerial vehicles, which is not the first unmanned aerial vehicle, acquires position information and information on the movement direction of the first unmanned aerial vehicle, and controls the position of the second unmanned aerial vehicle so that the relative position or relative distance of the second unmanned aerial vehicle to the position and movement direction of the first unmanned aerial vehicle satisfies a target value or target range.

10. An imaging system as described in claim 8, wherein the second unmanned aerial vehicle is determined to be in formation flight abnormality when the relative position or relative distance of the second unmanned aerial vehicle to the first unmanned aerial vehicle does not satisfy the target value or target range.

11. An imaging system as described in claim 8, wherein the second unmanned aerial vehicle is judged to be in formation flight abnormality when the relative speed of the second unmanned aerial vehicle with respect to the first unmanned aerial vehicle does not satisfy a target value or target range, or when the difference between the absolute speed of the second unmanned aerial vehicle and a target absolute speed exceeds a predetermined value.

12. An imaging system as described in claim 1, wherein the unmanned aerial vehicle in which a control communication abnormality or positioning accuracy abnormality is detected is determined to be suffering from a formation flight abnormality, or the first unmanned aerial vehicle in which an abnormality in the positioning accuracy of the subject to be photographed is detected is determined to be suffering from a formation flight abnormality.

13. A photography system as described in claim 1, wherein, when the first unmanned aerial vehicle is determined to be experiencing a formation flight abnormality, the entire formation consisting of the multiple unmanned aerial vehicles including the first unmanned aerial vehicle is caused to carry out the evacuation action.

14. A photography system as described in claim 8, wherein, when the second unmanned aerial vehicle is determined to be experiencing the formation flight abnormality, the second unmanned aerial vehicle experiencing the formation flight abnormality is caused to perform the evacuation action, or the entire formation consisting of the multiple unmanned aerial vehicles including the second unmanned aerial vehicle experiencing the formation flight abnormality is caused to perform the evacuation action.

15. An imaging system as described in claim 1, wherein the orientation of the camera of the unmanned aerial vehicle is controlled so that the subject to be photographed appears near the center of the image captured by the camera.

16. An imaging system as described in claim 3, wherein the position of the subject to be photographed is determined from a sensor provided on the subject, or the position of the subject to be photographed is determined by a sensor provided on the unmanned aerial vehicle, or the position of the subject to be photographed is determined by a sensor provided in the environment in which the subject to be photographed moves, or the position of the subject to be photographed is determined from the image captured by the camera.

17. An imaging system as described in claim 16, wherein the moving direction of the subject to be photographed is determined from a sensor provided on the subject, or the moving direction of the subject to be photographed is determined by a sensor provided on the unmanned aerial vehicle, or the moving direction of the subject to be photographed is determined by a sensor provided in the environment in which the subject to be photographed moves, or the moving direction of the subject to be photographed is determined from the image captured by the camera.

18. In the photography system described in claim 1, the formation flight positions of the multiple unmanned aerial vehicles include at least two of the following: a forward position in a forward direction relative to the subject to be photographed, a rear position in a rearward direction relative to the subject to be photographed, a right position in a rightward direction relative to the subject to be photographed, a left position in a leftward direction relative to the subject to be photographed, and a directly above position in a directly above direction relative to the subject to be photographed, photography system.

19. An imaging system as described in claim 18, wherein the evacuation action includes the following movements depending on the formation flight position; in the case of an unmanned aerial vehicle in the front position, including movement in a direction increasing altitude, movement to the left or right, or movement backwards; in the case of an unmanned aerial vehicle in the rear position, including movement in a direction increasing altitude, movement to the left or right, or movement backwards; in the case of an unmanned aerial vehicle in the right position, including movement in a direction increasing altitude, movement to the right, or movement backwards; in the case of an unmanned aerial vehicle in the left position, including movement in a direction increasing altitude, movement to the left, or movement backwards; and in the case of an unmanned aerial vehicle in the directly above position, including movement in a direction increasing altitude, movement to the left or movement backwards.

20. A method for photographing in an photography system using unmanned aerial vehicles, the photography system comprising a plurality of unmanned aerial vehicles flying in pursuit of a subject to be photographed, each unmanned aerial vehicle among the plurality of unmanned aerial vehicles photographing the subject to be photographed with a camera to provide photographed footage, the plurality of unmanned aerial vehicles flying in a formation flight position in a predetermined positional relationship with the subject to be photographed, the photography method comprising the steps of: controlling the flight of at least one unmanned aerial vehicle among the plurality of unmanned aerial vehicles so that at least one parameter value of the relative position, relative distance, absolute position, relative speed, and absolute speed with respect to the subject to be photographed satisfies a predetermined condition; determining that an unmanned aerial vehicle that does not satisfy the predetermined condition is engaged in formation flight abnormality; and moving the unmanned aerial vehicle engaged in formation flight abnormality in a direction away from the subject to be photographed.

21. A program for causing a computer to execute the processing by the imaging system according to claim 1.

Images