System for controlling mobile photographing device

Abstract

The present invention makes it possible to more appropriately photograph a subject moving on a predetermined course by using a mobile photographing device.　Provided is a system 300 for controlling a mobile photographing device 100 that has a camera with which a subject 400 moving on a course 10 is photographed while the device is moved. The system 300 controls the position of the mobile photographing device 100 on the basis of the position of the course 10, the position of an obstacle 20 around a course 30, and the position of the subject 400 such that, when viewed from the camera of the mobile photographing device 100, the subject 400 is prevented from being hidden in a blind spot behind the obstacle 20, or a blind spot does not occur where at least a part of the course 10 is hidden within the photographing field angle of the camera.

Description

System for controlling a mobile imaging device

[0001] The present invention relates to a system for controlling a mobile photographing device having a camera for photographing a subject moving within a course while the camera is moving.

[0002] For example, in racing sports where athletes race on a specific course, there is a need for a filming method that tracks and films athletes so that viewers can more easily grasp the race rankings and the movements and expressions of each athlete, etc. For this reason, filming using fixed cameras or cameramen installed at fixed locations along the course (e.g., in motor racing), or cameras running alongside the athletes on the course (e.g., in marathons), has been used for many years.

[0003] In the case of fixed cameras, the range that can be captured by a single fixed camera is limited on courses with poor visibility, so it is necessary to install many cameras along the course, which is costly. Furthermore, in order to reliably track a specific runner on video, it is necessary to frequently switch between the images from multiple fixed cameras. In the case of parallel cameras along the course, in marathons and bicycle races where athletes travel at relatively slow speeds on paved courses, it is possible to have a cameraman on a motorcycle or car travel alongside the runners. However, in events that use unpaved courses (such as skiing, yachting, and off-road motorcycle races) or events where athletes travel at high speeds (such as motorcycle and car races), it is difficult to have a cameraman travel alongside the athletes.

[0004] For this reason, in recent years, aerial photography using manned or unmanned aircraft equipped with cameras has become common (see, for example, Patent Document 1). When using such aircraft, a wide area of ​​the course can be photographed from above, and the photographing position can be moved while photographing, eliminating the need for as many cameras as in the case of fixed cameras. Furthermore, unlike cameras running alongside the course, photographing the course from above makes it possible to track and photograph objects even when the course surface is in poor condition or the speed at which the competing objects are moving.

[0005] Patent No. 6885485

[0006] However, the above technology has the following problems.

[0007] If there is a tunnel on the course, or trees, slopes, or buildings around the course, even when filming from above, athletes running on the course will sometimes be in the camera's blind spot, making it impossible to film them continuously.

[0008] The present invention has been made in view of the above circumstances, and has as its object to enable a subject moving on a predetermined course to be photographed more appropriately using a mobile photographing device.

[0009] According to one embodiment, a system for controlling a mobile photographing device having a camera for photographing a subject moving within a course while moving includes a course information acquisition unit that acquires course information including information about the position of the course, an obstacle information acquisition unit that acquires obstacle information including information about the positions of obstacles present around the course, a subject information acquisition unit that acquires subject information including information about the position of the subject, and a mobile control unit that controls the mobile photographing device based on the course information, the obstacle information, and the subject information so as to avoid the subject being hidden in a blind spot behind the obstacle as viewed from the camera of the mobile photographing device, or so as to avoid the occurrence of a course blind spot, which is a blind spot that hides at least a portion of the course within the photographing field of view of the camera.

[0010] According to one embodiment, a moving image capture device can be used to better capture images of a subject moving within a course.

[0011] FIG. 1 is a configuration diagram of an entire system including a photography control system according to one embodiment. FIG. 2 is a system configuration diagram of an unmanned aerial vehicle according to one embodiment. FIG. 3 is a schematic diagram showing the exterior of an unmanned aerial vehicle according to one embodiment. FIG. 4 is a system configuration diagram of a mobile camera according to one embodiment. FIG. 5 is a schematic diagram showing the exterior of a mobile camera according to one embodiment. FIG. 6 is a system configuration diagram of a control device according to one embodiment. FIG. 7 is a schematic diagram showing the exterior of a control device according to one embodiment. FIG. 8 is a schematic diagram showing types of movement methods for an unmanned aerial vehicle 100. FIG. 9 is a system configuration diagram of an athlete's device according to one embodiment. FIG. 10 is a physical configuration diagram of a server according to one embodiment. FIG. 11 is a system configuration diagram of a server according to one embodiment. FIG. 12 is a flowchart showing the overall control operation of the server according to one embodiment. FIG. 13 is a plan view showing the positions and shapes of a course and obstacles in a plan view according to one embodiment. FIG. 14 is a side view showing the positions and shapes of a course and obstacles in a side view according to one embodiment. FIG. 15 is a plan view showing the positions and shapes of a flight geofence and a flight path in a plan view according to one embodiment. FIG. 16 is a side view showing the position and shape of a flight geofence and a flight path in a side view according to one embodiment. FIG. 17 is an explanatory diagram showing a process for acquiring camera information according to one embodiment. FIG. 18 is an explanatory diagram showing a process for determining a communication anomaly according to one embodiment. FIG. 19 is an explanatory diagram showing a process for estimating a blind spot position according to one embodiment. FIG. 20 is an explanatory diagram showing a process for acquiring subject information according to one embodiment. FIG. 21 is an explanatory diagram showing a process for determining a positional relationship according to one embodiment. FIG. 22 is an explanatory diagram showing a process for generating a movement plan according to one embodiment. FIG. 23 is an explanatory diagram showing a process for generating a movement plan according to one embodiment. FIG. 24 is an explanatory diagram showing a process for generating a movement plan according to one embodiment. FIG. 25 is an explanatory diagram showing a process for generating a movement plan according to one embodiment. FIG. 26 is an explanatory diagram showing a process for generating a movement plan according to one embodiment. FIG. 27 is an explanatory diagram showing a process for generating a movement plan according to one embodiment. FIG. 28 is an explanatory diagram showing a process for generating a movement plan according to one embodiment. FIG. 29 is an explanatory diagram showing a process for generating a movement plan according to one embodiment. FIG. 30 is a flowchart showing a determination process of a command generation unit according to one embodiment.FIG. 31 is a flowchart showing the determination process of the camera control command unit according to one embodiment.

[0012] The following description of the embodiments will be made with reference to the drawings. Note that the embodiments described below do not limit the scope of the invention as claimed, and not all of the elements and combinations thereof described in the embodiments are necessarily essential to the solution of the invention.

[0013] 1 is a diagram showing the configuration of an entire system including an imaging control system for controlling a mobile imaging device according to an embodiment. In this embodiment, the imaging control system is embodied as a server 300 including a programmed computer system, for example.

[0014] Here, for illustrative purposes, a scene of a car race will be assumed, in which one or more vehicles 400 (subjects) are traveling on a road 10 (i.e., a course) in a mountainous region as shown in the figure. An unmanned aerial vehicle 100 (i.e., a mobile imaging device) flies in the area above the road 10 around the vehicle 400. The unmanned aerial vehicle 100 is an aircraft without a human on board (a so-called drone), and is, for example, battery-powered. The unmanned aerial vehicle 100 is equipped with a camera (not shown), and while flying and tracking the vehicle 400 traveling on the road 10, it photographs the vehicle 400 with the onboard camera. In addition, a mobile camera 710 that photographs the vehicle 400 may also be installed on the ground. Unmanned aerial vehicle 100 and mobile camera 710 are each remotely controlled by a control device 200 operated by, for example, a ground-based pilot 30 (although only one control device 200 is shown, typically unmanned aerial vehicle 100 and mobile camera 710 are each controlled by a separate control device 200). At various points along the side of road 10, there are obstacles 20, such as trees, that may obstruct filming by vehicle 400.

[0015] Each of unmanned aerial vehicle 100 and vehicle 400 can communicate with a plurality of satellites 500, such as a Global Navigation Satellite System (GNSS), and receives satellite signals transmitted from satellites 500 to determine its own position. A reference position communication station 600 may be provided at the race site. Reference position communication station 600 is also capable of communicating with satellites 500. Reference position communication station 600 provides pilot device 200 with information about the local reference position of the site to improve the accuracy of control of the position of unmanned aerial vehicle 100.

[0016] 1 shows one each of unmanned aerial vehicle 100, mobile camera 710, control device 200, and vehicle 400 for convenience of illustration, however, in reality, there may be multiple unmanned aerial vehicles 100, multiple mobile cameras 710, multiple control devices 200, and multiple vehicles 400 at the site. The following description focuses on the single unmanned aerial vehicle 100, mobile camera 710, control device 200, and vehicle 400 shown in the figure.

[0017] The server 300 may be located anywhere inside or outside the site and can communicate with the vehicle 400, the control device 200, and the display device 900, for example, via a wireless communication network 800. The server 300 may be able to communicate with one or more information sources (not shown), such as a geographic database that provides geographic coordinate information indicating the three-dimensional position coordinates (e.g., latitude, longitude, and elevation) of multiple locations in a geographic area including the course 10 and obstacles 20. The server 300 acquires vehicle information, including information regarding the three-dimensional position of the vehicle 400, from the vehicle 400, and acquires information, such as information regarding the three-dimensional positions, camera orientations, and predetermined states, from the unmanned aerial vehicle 100 and the mobile camera 710. The server 300 acquires and stores in advance course information, including information regarding the three-dimensional position of the course 10, and obstacle information, including information regarding the three-dimensional positions of obstacles, by communicating with the geographic database or by manual input. Based on this various information, server 300 generates commands for controlling the position and orientation of unmanned aerial vehicle 100 and mobile camera 710 and transmits them to control device 200. Although one server 300 is shown in FIG. 1 , multiple servers 300 may be present. However, the following description focuses on one server 300. Here, the above-mentioned "information related to position" may include not only position (two-dimensional and / or three-dimensional position) but also physical quantities related to position, such as velocity and / or acceleration.

[0018] The images captured by the unmanned aerial vehicle 100 and the mobile camera 710 are edited into broadcast images by the server 300, and the broadcast images are transmitted to the display device 900 and provided to spectators 40 (e.g., general viewers, race commentators, etc.).

[0019] Fig. 2 is a system configuration diagram of the unmanned aerial vehicle 100 according to one embodiment. Fig. 3 is a schematic diagram showing the appearance of the unmanned aerial vehicle 100 according to one embodiment.

[0020] The unmanned aerial vehicle 100 includes a photographing unit 110 , a flight function unit 120 , an obstacle detection unit 130 , an aircraft state acquisition unit 140 , an abnormality detection unit 150 , and a communication unit 160 .

[0021] The photographing unit 110 includes a photographing camera 111 , a camera holding unit 112 , a camera control unit 113 , and a holding control unit 114 .

[0022] The photographing camera 111 is attached to, for example, the exterior surface of the unmanned aerial vehicle 100, and photographs the outside world from its attachment position on the unmanned aerial vehicle 100 in any three-dimensional direction (for example, up and down, left and right, or a diagonal direction between these) within a certain variable angle range. In the example shown in Figure 3, the photographing camera 111 is attached to the side of the housing of the unmanned aerial vehicle 100 via a camera holder 112.

[0023] The camera holding unit 112 holds the photographing camera 111 so that its orientation can be changed, and is configured, for example, by a gimbal whose rotation axis is configured by a motor. In the example shown in Figure 3, the camera holding unit 112 is attached to the side of the housing of the unmanned aerial vehicle 100.

[0024] The camera control unit 113 is composed of a calculation device such as a CPU, and executes various control programs for controlling the photographing camera 111 in response to control commands from the control device 200. The camera control unit 113 executes programs for performing, for example, zoom control, vibration reduction control, automatic exposure control, automatic focus control, and subject detection.

[0025] The holding control unit 114 is composed of an arithmetic device such as a CPU, and outputs a drive signal to the camera holding unit 112 so as to point the photographing camera 111 held by the camera holding unit 112 toward a desired subject, such as a vehicle 400 running within the course 10.

[0026] The flight function unit 120 includes a plurality of motors 121 , a plurality of propellers 122 , and a flight control unit 123 .

[0027] The plurality of motors 121 rotate the plurality of propellers 122. In the example shown in Fig. 3, one motor 121 is provided for one propeller 122.

[0028] The multiple propellers 122 generate thrust for flying the unmanned aerial vehicle 100. In consideration of the balance between flight stability, airframe size, and power consumption, in the example shown in Figure 3, the propellers 122 are provided at four locations on the unmanned aerial vehicle 100, but they may be provided at more locations.

[0029] The flight control unit 123 is composed of a calculation device such as a CPU, and drives the motor 121 based on control commands from the control device 200 to control the flight and takeoff and landing of the unmanned aerial vehicle 100. If the flight control unit 123 detects an obstacle that may interfere with the flight of the unmanned aerial vehicle 100 based on the detection results of the obstacle detection sensor 131, it causes the unmanned aerial vehicle 100 to hover, turn back, or take a detour.

[0030] The obstacle detection unit 130 includes an obstacle detection sensor 131 and a detection processing unit 132 .

[0031] The obstacle detection sensor 131 detects obstacles 20 present around the unmanned aerial vehicle 100 and the portion of the course 10 to be photographed. The obstacle detection sensor 131 is configured using, for example, an optical camera, a Time of Flight (ToF) sensor, a Laser Imaging Detection and Ranging (LIDAR), or an infrared (IR) sensor. In the example shown in Fig. 3, two optical cameras are provided as obstacle detection sensors 131 at the front of the unmanned aerial vehicle 100, but they may also be provided not only on the forward side but also on other sides or on all six sides.

[0032] The detection processing unit 132 is configured with a calculation device such as a CPU, and determines the presence, position, shape, etc. of an obstacle 20 based on sensing information from the obstacle detection sensor 131.

[0033] The aircraft state acquisition unit 140 includes an aircraft position measurement unit 141 , an aircraft orientation measurement unit 142 , an aircraft speed measurement unit 143 , and a zoom amount acquisition unit 144 .

[0034] The aircraft's own position measurement unit 141 includes a GNSS receiver such as a GPS (Global Positioning System) receiver, and uses the GNSS signals received by the GNSS receiver to measure the three-dimensional position (e.g., latitude, longitude, and altitude) of the unmanned aircraft 100. The aircraft's own position measurement unit 141 may also be able to measure the position of the unmanned aircraft 100 using autonomous navigation based on the output of a standalone positioning sensor such as a gyro sensor.

[0035] The aircraft's own orientation measurement unit 142 is configured using, for example, a three-axis compass (three-axis geomagnetic sensor) and measures the orientation of the unmanned aircraft 100 on Earth.

[0036] The aircraft speed measurement unit 143 measures and outputs the speed (magnitude and direction) of the unmanned aircraft 100 in the direction of movement while in flight, for example, based on the position measured by the aircraft position measurement unit 141, or using a gyro sensor, accelerometer, or optical flow sensor.

[0037] The zoom amount acquisition unit 144 acquires the current zoom amount of the photographing camera 111 via the camera control unit 113 .

[0038] The abnormality detection unit 150 includes a control communication abnormality detection unit 151 and a positioning state abnormality detection unit 152 .

[0039] The control communication abnormality detection unit 151 determines that the communication status between the unmanned aircraft 100 and the control device 200 is abnormal if a predetermined time period has passed during which the communication unit 160 has not received a control signal from the control device 200.

[0040] The positioning state abnormality detection unit 152 detects that there is an abnormality in the positioning state when the communication unit 160 cannot normally receive radio waves from the artificial satellites 500 or the reference position communication station 600, or when there are an insufficient number of artificial satellites 500 with which it can communicate.

[0041] The communication unit 160 performs wireless communication with the control device 200, the artificial satellite 500, and the reference position communication station 600. Note that the communication unit 160 may also perform wireless communication with other unmanned aerial vehicles 100.

[0042] Fig. 4 is a system configuration diagram of a mobile camera 710 according to an embodiment. Fig. 5 is a schematic diagram showing the appearance of the mobile camera 710 according to an embodiment.

[0043] The mobile camera 710 includes an imaging unit 7110 , a driving unit 7120 , a status acquisition unit 7130 , an abnormality detection unit 7140 , and a communication unit 7150 .

[0044] The photographing unit 7110 includes a photographing camera 7111 and a camera control unit 7112 .

[0045] The photographing camera 7111 photographs the outside world from any position within a certain movable range of the mobile camera 710, facing any direction within a certain direction variable range (for example, up and down, left and right, or a diagonal direction between these). In the example shown in FIG. 5, the photographing camera 7111 is provided so as to be movable along a guide rail 7124.

[0046] The camera control unit 7112 is composed of a calculation device such as a CPU, and executes various control programs for controlling the photographing camera 7111 in response to control commands from the control device 200. The camera control unit 7112 executes programs for performing, for example, zoom control, vibration reduction control, automatic exposure control, automatic focus control, and subject detection.

[0047] The drive unit 7120 includes a camera orientation adjustment unit 7121 , a drive control unit 7122 , a camera position adjustment unit 7123 , and a guide rail 7124 .

[0048] The camera orientation adjustment unit 7121 controls the orientation of the photographing camera 7111. The drive control unit 7122 drives and controls the camera orientation adjustment unit 7121 and the camera position adjustment unit 7123. The camera position adjustment unit 7123 controls the position of the photographing camera 7111 along the guide rail 7124. The guide rail 7124 is installed, for example, in a predetermined section around the course 10 so that a predetermined portion of the course 10 can be photographed.

[0049] The status acquisition unit 7130 includes a camera position acquisition unit 7131 , a camera orientation acquisition unit 7132 , a speed measurement unit 7133 , and a zoom amount acquisition unit 7134 .

[0050] The camera position acquisition unit 7131 measures and acquires the position of the photographic camera 7111. The camera orientation acquisition unit 7132 measures and acquires the orientation of the photographic camera 7111. The own aircraft speed measurement unit 7133 measures and acquires the movement speed of the photographic camera 7111. The zoom amount acquisition unit 7134 acquires the zoom amount of the photographic camera 7111 via the camera control unit 7112.

[0051] The abnormality detection unit 7140 includes a control communication abnormality detection unit 7141 and a positioning state abnormality detection unit 7142 .

[0052] The control communication abnormality detection unit 7141 determines that the communication status between the mobile camera 710 and the control device 200 is abnormal if a predetermined time period has passed during which the communication unit 7150 has not received a control signal from the control device 200.

[0053] The positioning status abnormality detection unit 7142 detects that there is an abnormality in the positioning status when the communication unit 7150 cannot properly receive radio waves from the artificial satellites 500 or the reference position communication station 600, or when there are insufficient artificial satellites 500 with which to communicate.

[0054] The communication unit 7150 performs wireless communication with the control device 200, the artificial satellite 500, and the reference position communication station 600. Note that the communication unit 7150 may also perform wireless communication with other mobile cameras 710.

[0055] 6 is a system configuration diagram of a control device 200 according to one embodiment. In this embodiment, not only the control device 200 for remotely controlling the unmanned aerial vehicle 100, but also another control device 200 for remotely controlling the mobile camera 710 may be used. However, the following description focuses only on the control device 200 for the unmanned aerial vehicle 100, and describes its configuration and functions. From this description, those skilled in the art should be able to easily understand the control device 200 for the mobile camera 710 as well.

[0056] The control device 200 includes a display unit 210 and an input unit 220 .

[0057] The display unit 210 includes a control command display unit 211 and a shooting state display unit 212 .

[0058] The control command display unit 211 displays the control commands input from the server 300. The image capture status display unit 212 displays the images captured by the unmanned aerial vehicle 100 and the associated status such as orientation and position. The image capture status display unit 212 also displays an image showing how the unmanned aerial vehicle 100 is controlled by the control commands input from the server 300.

[0059] The input unit 220 includes a flight position operation unit 221 , a flight attitude operation unit 222 , a camera attitude operation unit 223 , a camera zoom operation unit 224 , a flight mode switching unit 225 , and a power input unit 226 .

[0060] The flight position operation unit 221 operates the position of the unmanned aerial vehicle 100. The flight attitude operation unit 222 operates the attitude (three-dimensional direction) of the unmanned aerial vehicle 100. The camera attitude operation unit 223 operates the attitude (three-dimensional direction) of the photographing camera 111. The camera zoom operation unit 224 operates the zoom amount of the photographing camera 111. The flight mode switching unit 225 switches between manual control mode and autopilot mode. Furthermore, in autopilot mode, the flight mode switching unit 225 can selectively execute escape flight control such as hovering, return, or soft landing on the spot. The power input unit 226 receives operation instructions from the pilot and inputs power to the unmanned aerial vehicle 100.

[0061] 7 is a schematic diagram showing the appearance of the control device 200 according to one embodiment. FIG. 8 is a schematic diagram showing the types of movement methods for the unmanned aerial vehicle 100.

[0062] 7 shows a smartphone as an example of the display unit 210. The display unit 210 may be, for example, a smartphone, a tablet terminal, or a dedicated display panel. The control device 200 is used while being held in both hands by the operator using the control device 200, for example.

[0063] The control device 200 has, for example, a housing 220 that is approximately rectangular parallelepiped in shape. A left control rod 63 and a right control rod 53 protrude from the surface of the housing 220 of the control device 200. As shown in Figures 7 and 8 , the left control rod 63 is used to remotely control the forward movement 60, reverse movement 62, left turn 64, and right turn 61 of the unmanned aerial vehicle 100. The right control rod 53 is used to remotely control the ascent 50, descent 54, left movement 55, and right movement 52 of the unmanned aerial vehicle 100. Furthermore, a button 51 for operating panning to change the angle of view of the photographing camera 111 and a button 65 for operating zooming are arranged on the side of the housing of the control device 200.

[0064] FIG. 9 is a system configuration diagram of the athlete's side device 450 according to one embodiment.

[0065] The athlete's device 450 includes a status acquisition unit 460, a photographing unit 470, an abnormality detection unit 480, and a communication unit 490. The athlete's device 450 is mounted on the vehicle 400 and / or the helmet of the driver 410 of the vehicle 400.

[0066] The status acquisition unit 460 includes a position measurement unit 461, a direction measurement unit 462, and a speed measurement unit 463 that respectively measure the current position, direction, and speed of the vehicle 400. The status acquisition unit 460 can use GNSS, an IMU (Inertial Measurement Unit), a transponder, or the like to measure the current position, direction, and speed of the vehicle 400. When a transponder is used, for example, transmitters (or receivers) are installed at predetermined intervals (for example, 100 m intervals) on the course 10, and the receivers (or transmitters) are mounted on the vehicle 400, thereby making it possible to measure the position, direction, and speed of the vehicle 400.

[0067] The photographing unit 470 includes an onboard camera 471 that photographs the outside world from the position of the vehicle 400. The onboard camera 471 may also be controlled by the server 300, similar to the photographing camera 111 of the unmanned aerial vehicle 100 (the control method should be easily understood by those skilled in the art from the explanation of the control method for the photographing camera 111 described in this specification).

[0068] The abnormality detection unit 480 includes a positioning state abnormality detection unit 481. The positioning state abnormality detection unit 481 detects that there is an abnormality in the positioning state when the communication unit 490 cannot normally receive radio waves from the artificial satellite 500 or the reference position communication station 600.

[0069] The communication unit 490 performs wireless communication with the artificial satellite 500 and the reference position communication station 600 .

[0070] FIG. 10 is a diagram showing the physical configuration of the server 300 according to an embodiment.

[0071] 10 , server 300 has a computer system 310, which includes a communication device 301, a storage device 302, and a processing device 303 connected thereto. Communication device 301 communicates with vehicle 400, control device 200 of unmanned aerial vehicle 100, and display device 900. Storage unit 302 stores and holds various computer programs (not shown) that can be executed by processing device 303. Processing device 303 reads and executes the computer programs from storage unit 302, thereby performing various control functions of server 300 described below.

[0072] FIG. 11 is a system configuration diagram of the server 300 according to an embodiment.

[0073] The server 300 includes a course information acquisition unit 310, a flight area setting unit 320, a camera information acquisition unit 330, an abnormal state information acquisition unit 340, a blind spot estimation unit 350, a subject information acquisition unit 360, a positional relationship determination unit 370, a camera control command generation unit 380, and a recording unit 390.

[0074] The course information acquisition unit 310 includes a course position acquisition unit 311 and an obstacle position acquisition unit 312 .

[0075] The course position acquisition unit 311 acquires course information including information on the three-dimensional positions and shapes of the course 10 (for example, position coordinate information of many points on the course 10). The obstacle position acquisition unit 312 acquires obstacle information including information on the three-dimensional positions and shapes of each obstacle (tree, building, structure, cliff, etc.) 20 (for example, position coordinate information of many points on the obstacle 200).

[0076] The flight area setting unit 320 includes a geofence setting unit 321 and a flight path setting unit 322 .

[0077] The geofence setting unit 321 sets a geofence (which may be a linear flight permission path) that defines the spatial area in which unmanned aerial vehicle 100 is permitted to fly, in a three-dimensional spatial area where unmanned aerial vehicle 100 can fly safely without risk of collision with obstacle 200. In this embodiment, the spatial area defined by the geofence is called a "flight area." The flight area may be set as a three-dimensional space, a two-dimensional plane, or a one-dimensional flight permission path (target path).

[0078] The flight path setting unit 322 sets a flight path within the range of the geofence set by the geofence setting unit 321.

[0079] The camera information acquisition unit 330 includes a camera state acquisition unit 331 and a camera movement prediction unit 332 .

[0080] The camera status acquisition unit 331 acquires the current status (e.g., position, orientation, speed, zoom amount, etc.) of each of the photographic cameras 111, 7110 from the unmanned aerial vehicle 100 and the mobile camera 710. The camera movement prediction unit 332 predicts the expected position of each photographic camera 111, 7110 at one or more future times (e.g., a single point in time several seconds from the present, or multiple points in time several seconds apart) based on the current status of each photographic camera 111, 7110 and information on the flight area. Here, although the position of the unmanned aerial vehicle 100 and the position of its photographic camera 111 at a certain time are technically different, in practice, they can often be considered to be the same. Therefore, the following description will be given assuming that the position of the unmanned aerial vehicle 100 and the position of the photographic camera 111 are the same.

[0081] The abnormality state information acquisition unit 340 determines whether or not there is a control communication abnormality and whether or not there is a positioning communication abnormality based on the detection results acquired from the abnormality detection unit 150 of the unmanned aerial vehicle 100 and the abnormality detection unit 7140 of the mobile camera 710.

[0082] The blind spot estimation unit 350 includes a future blind spot estimation unit 351 and a current blind spot determination unit 352 .

[0083] The future blind spot estimation unit 351 estimates one or both of the following two items regarding the blind spot as viewed from each estimated position of each photographic camera 111, 7110 at each future time, based on the estimated positions of each photographic camera 111, 7110 at each future time predicted by the camera movement prediction unit 332 and the position information of the obstacle 20 and the position information of the course 10 obtained from the obstacle information and course information described above. The first item regarding the blind spot is a phenomenon in which at least a part of the course 10 is hidden in a blind spot caused by an obstacle 20 as viewed from each photographic camera 111, 7110. Here, a blind spot that hides at least a part of the course 10 is referred to as a "course blind spot." Estimating the first item is determining whether or not a course blind spot has occurred. The second item regarding the blind spot is the position of the blind spot caused by the obstacle 20 (or, among the blind spots, the above-mentioned course blind spot in particular) as viewed from each photographic camera 111, 7110. This second item will be referred to as the "blind spot position" hereinafter. When the onboard camera 471 on the vehicle 400 or the driver 410 is used as a photographing camera, the position and / or attitude (direction) of the vehicle 400 or the driver 410 at each future time is predicted, and based on the prediction, the presence or absence of a blind spot on the course as seen from the onboard camera 471 and / or the location of the blind spot are estimated.

[0084] The current blind spot determination unit 352 determines whether or not a blind spot currently exists on the course and / or the location of the blind spot based on the current position and orientation of each photographing camera 111, 7110, as well as the position information of the course 10 and the position information of the obstacle 20 described above.

[0085] The subject information acquisition unit 360 includes a current state acquisition unit 361 and a future prediction unit 362 .

[0086] The current state acquisition unit 361 acquires the current position, speed, and presence or absence of positioning abnormalities (e.g., satellite signal reception status) of the vehicle 400 (subject). The current state acquisition unit 361 wirelessly acquires current position information (e.g., latitude, longitude, altitude, and speed) of the vehicle 400 from, for example, a GNSS receiver mounted on the vehicle 400 (subject). Alternatively, the current state acquisition unit 361 may estimate the current position of the vehicle 400 based on, for example, video of the vehicle 400 captured by the unmanned aerial vehicle 100 and the position information of the unmanned aerial vehicle 100 at the time of capturing the video. Alternatively, the current state acquisition unit 361 may measure the current position of the vehicle 400 using, for example, a sensor (camera, LiDAR, IR, radar, etc.) of an external device other than the unmanned aerial vehicle 100. Alternatively, the current state acquisition unit 361 may measure the current position of the vehicle 400 using a transponder such as that described above.

[0087] The future prediction unit 362 predicts the expected position (e.g., latitude, longitude, altitude, and speed) of the vehicle 400 (subject) at each of one or more future times (e.g., at intervals of several seconds). The future prediction unit 362 predicts the expected position of the vehicle 400 at each future time, for example, based on current position information of the vehicle 400. Alternatively, the future prediction unit 362 may predict the expected position of the vehicle 400 at each future time, for example, based on statistical information such as movement statistics (performance statistics) of the vehicle 400 in multiple past races. Alternatively, the future prediction unit 362 may predict the expected position of the vehicle 400 at each future time by combining the current position of the vehicle 400 with the above statistical information. Alternatively, the future prediction unit 362 may predict the expected position of the vehicle 400 based on a timetable submitted in advance (e.g., performance results in race qualifying).

[0088] Positional relationship determination unit 370 includes a future positional relationship estimation unit 371 and a current positional relationship determination unit 372 .

[0089] The future positional relationship estimation unit 371 compares, for each future time, the presence or absence of a blind spot on the course for each photographic camera 111, 7110 or the position of the blind spot with the expected three-dimensional position of the vehicle 400, and determines whether the vehicle 400 will be hidden in a blind spot as viewed from each photographic camera 111, 7110. The future positional relationship estimation unit 371 may determine that the vehicle 400 will be hidden in a blind spot on the condition that the current position of the vehicle 400 is closer to the blind spot position by a predetermined threshold or more and the speed is higher by a predetermined threshold or more. The current positional relationship determination unit 372 performs a similar comparison with respect to the current time to determine whether the vehicle 400 will currently be hidden in a blind spot as viewed from each photographic camera 111, 7110.

[0090] The camera control command generation unit 380 includes a command generation unit 381 (flight control unit) and a command display unit 382 .

[0091] The command generation unit 381 generates control commands for controlling each of the unmanned aerial vehicle 100 and the mobile camera 710 and transmits them to the respective control devices 200 of the unmanned aerial vehicle 100 and the mobile camera 710. The respective control devices 200 of the unmanned aerial vehicle 100 and the mobile camera 710 receive the control commands and control the movement and shooting operations of the unmanned aerial vehicle 100 and the mobile camera 710 in accordance with the received control commands. For example, in the case of the unmanned aerial vehicle 100, a control command from the server 300 is received by the control device 200, and the control device 200 automatically (or semi-automatically with manual control added) flies the unmanned aerial vehicle 100 in accordance with the control command. In addition, the display unit 210 of the control device 200 of the unmanned aerial vehicle 100 displays an image showing how the unmanned aerial vehicle 100 will operate in accordance with the control command.

[0092] When the positional relationship determination unit 370 determines, for each photographic camera 111, 7110, that vehicle 400 (subject) is hidden in a blind spot (this may include not only cases where the entire subject is in the blind spot, but also cases where only a portion of the subject is in the blind spot), the command generation unit 381 generates time-series target values ​​for the position, orientation, and time of each photographic camera 111, 7110 that will prevent vehicle 400 from being hidden in the blind spot, based on the estimation results by the blind spot estimation unit 350, and generates movement plans for unmanned aerial vehicle 100 and mobile camera 710 using these time-series target values. Each movement plan is transmitted as a control command to the control device 200 of each unmanned aerial vehicle 100 and mobile camera 710. After a movement plan has been generated, the command generation unit 381 changes the movement plan if the positional relationship determination unit 370 determines that vehicle 400 is hidden in a blind spot. Changes to the movement plans of unmanned aerial vehicle 100 and mobile camera 710 are transmitted as control commands to the control devices 200 of unmanned aerial vehicle 100 and mobile camera 710, respectively. Specific examples of changes to the movement plan may include changing the speed of unmanned aerial vehicle 100, increasing the altitude of unmanned aerial vehicle 100, changing the flight path of unmanned aerial vehicle 100, moving unmanned aerial vehicle 100 closer to or farther away from the course, etc. Note that the movement plan for unmanned aerial vehicle 100 may be referred to as a "flight plan" in the following description.

[0093] When the positional relationship determination unit 370 determines that the vehicle 400 (subject) is hidden in a blind spot, the command generation unit 381 can allocate the photographing work (hereinafter referred to as the task) of tracking and photographing the vehicle 400 in accordance with the time-series target values ​​described above to a plurality of unmanned aerial vehicles 100. For example, when one unmanned aerial vehicle 100 cannot realize the generated flight plan (time-series target values) (for example, when the flight speed performance of the unmanned aerial vehicle 100 is insufficient), the command generation unit 381 can transmit a control command for tracking standby, including the flight plan, to another unmanned aerial vehicle 100 flying at a different position around the course 10, and hand over the task of tracking flight and photographing in accordance with the flight plan to the other unmanned aerial vehicle 100.

[0094] Based on the control command (particularly the control command for the unmanned aircraft 100) generated by the command generation unit 381, the command display unit 382 generates an image to show the pilot how the unmanned aircraft 100 will be controlled by the control command, and transmits the image to the control device 200 to display it on its display unit 210.

[0095] The recording unit 390 includes a course position recording unit 391 , an obstacle position recording unit 392 , and a blind spot position recording unit 393 .

[0096] The course position recording unit 391 records and stores course information including information on the position and shape of the course acquired by the course position acquisition unit 311. The obstacle position recording unit 392 records and stores course information including information on the position and shape of the obstacle 20 acquired by the obstacle position acquisition unit 312. The blind spot position recording unit 393 records and stores the current and future blind spot positions of each of the photographing cameras 111, 7110 estimated by the blind spot estimation unit 350.

[0097] The control operation of server 300 will be described below. The following description focuses on the control of unmanned aerial vehicle 100. From this description, a person skilled in the art should be able to easily understand the control of mobile camera 710 by analogy.

[0098] FIG. 12 is a flowchart showing the overall control operation of the server 300 according to an embodiment.

[0099] In step S101, the course information acquisition unit 310 acquires course information including information indicating the three-dimensional position and shape of the course 10. The course 10 is, for example, a road in Japan having a position and shape in a plan view as shown in FIG. 13 and a position and shape in a side view as shown in FIG. 14. In this case, the above course information can be acquired by accessing an open-source geographic database made public by a public organization such as the Geospatial Information Authority of Japan, and the course information is recorded in the course position recording unit 391. Alternatively, for example, the course information acquisition unit 310 may acquire, as course information, a three-dimensional model of the course 10 (e.g., three-dimensional point cloud data) created in advance based on an image of the course 10 captured by the unmanned aerial vehicle 100, an artificial satellite 500, or the like before the start of the race. Alternatively, for example, the course information acquisition unit 310 may execute, for example, SfM (Structure from Motion) analysis software to perform topography calculations to generate a three-dimensional model of the course 10 from an image of the course 10 taken by the unmanned aerial vehicle 100, and use this as course information. Alternatively, the course information acquisition unit 310 may acquire, as course information, the three-dimensional position and three-dimensional shape of the course 10 that have been obtained in advance by actually driving a vehicle equipped with a positioning sensor on the course 10.

[0100] In step S102, the obstacle position acquisition unit 312 acquires obstacle information including information indicating the three-dimensional positions and shapes of obstacles 20 (trees, buildings, structures, cliffs, etc.) around the course 10, as illustrated in FIGS. 13 and 14B. For obstacles 20 that do not change over time, such as buildings, structures, or topography, the obstacle position acquisition unit 312 may acquire the three-dimensional positions and three-dimensional shapes of the obstacles 20 around the course as obstacle information from open-source geographic data published by, for example, the Geospatial Information Authority of Japan, and record this in the obstacle position recording unit 392. Alternatively, the obstacle position acquisition unit 312 may acquire, as obstacle information, a three-dimensional model (e.g., three-dimensional point cloud data) of the obstacle 20 that has been generated in advance from images of the area around the course taken by the unmanned aerial vehicle 100, the artificial satellite 500, or the like before the start of the race. Alternatively, the obstacle position acquisition unit 312 may execute SfM using images photographed from an aerial camera to generate a three-dimensional model of the obstacle 20, which may be used as obstacle information. In particular, for obstacles whose position or shape changes over time, such as trees, it is desirable to identify their position and shape using the results of photography and measurement by the unmanned aerial vehicle 100 or the artificial satellite 500.

[0101] In step S103, the flight area setting unit 320 sets a geofence 80 and a flight path 81 having a three-dimensional position and shape as exemplified in the plan view of FIG. 15 and the side view of FIG. 16. The geofence setting unit 321 of the flight area setting unit 320 sets a geofence (flight area) 80 in a space that does not interfere with the obstacle 20 and does not contact the vehicle 400 on the course 20, for example, based on information on the three-dimensional position and shape of the obstacle 20 and information on the position and shape of the course 10. The flight path setting unit 322 of the flight area setting unit 320 sets a flight path 81 within the range of the geofence (flight area) 80 set by the geofence setting unit 321. The flight path 81 is set at a safe altitude, for example, around 15 m above ground. The flight path setting unit 322 sets the flight path 81 within the geofence 80 where safety is ensured, at a position that minimizes the blind spots of the obstacles 20 that hide the vehicle 400 on the course 10.

[0102] In step S104, the camera state acquisition unit 331 acquires the current state of the photographic camera 111 (hereinafter referred to as the camera state) from the unmanned aerial vehicle 100. This camera state includes information such as the position, orientation, speed, and zoom amount of the photographic camera 111 acquired by the aircraft state acquisition unit 140 of the unmanned aerial vehicle 100.

[0103] The camera movement prediction unit 332 predicts the expected position of the imaging camera 111 at one or more future times based on the current camera state and information on the flight area 80. For example, as shown in FIG. 17 , the camera movement prediction unit 332 predicts expected positions A1 to A3 of the unmanned aerial vehicle 100 at future times t1 to t3 based on the position A0 and flight speed of the unmanned aerial vehicle 100 at the current time t0 and the flight path 81. The camera movement prediction unit 332 may predict the future expected position of the unmanned aerial vehicle 100 using past information, such as the movement history of the unmanned aerial vehicle 100 up to the present. Alternatively, the expected position may be predicted by combining the current position and speed with past information. Alternatively, the expected position may be predicted based on a time schedule, such as a flight plan prepared in advance.

[0104] In step S105, the abnormal condition information acquisition unit 340 acquires the detection results from the abnormality detection unit 150 of the unmanned aerial vehicle 100. Based on the acquired detection results, the abnormal condition information acquisition unit 340 determines whether or not there is an abnormality in (1) the control communication 82 and (2) the positioning communication 83 of the unmanned aerial vehicle 100, as shown in FIG.

[0105] In step S106, the future blind spot estimation unit 351 estimates whether or not a blind spot will occur on the course and / or the position of the blind spot at each predicted position of the unmanned aircraft 100 based on the predicted position of the unmanned aircraft 100 at each future time predicted by the camera movement prediction unit 332, the position of the obstacle 20, and the position of the course 10.

[0106] For example, as shown in Figure 19, at estimated positions A1 and A2 of unmanned aerial vehicle 100 predicted by camera movement prediction unit 332, there is no obstacle 20 between unmanned aerial vehicle 100 and the portion of course 10 that falls within the photographic field of view 90, so no blind spot (course blind spot) that hides course 10 occurs. However, at estimated position A3, there is an obstacle 20 between unmanned aerial vehicle 100 and the portion of course 10 that falls within the photographic field of view 90, so a course blind spot (e.g., elliptical zone 91 in Figure 19) occurs. Based on the data for each of the camera's estimated positions A1 to A3, the position data for the portion of course 10 that falls within the photographic field of view 90 at each of estimated positions A1 to A3, and the position data for obstacles 20 around that course 10, future blind spot estimation unit 351 estimates whether or not a course blind spot (e.g., elliptical zone 91 in Figure 19) will occur at each of estimated positions A1 to A3, and / or the position of the blind spot (e.g., the position of elliptical zone 91 in Figure 19).

[0107] The estimation of the occurrence of a blind spot on the course and the position of the blind spot may be calculated in real time based on the position of obstacle 20, the position of course 10, and the expected position of unmanned aerial vehicle 100. Alternatively, the occurrence of a blind spot on the course and the position of the blind spot at each position on flight path 81 of unmanned aerial vehicle 100 may be calculated in advance based on the position of obstacle 20 and the position of course 10 and stored in a database, and server 300 may use this database.

[0108] In step S107, the current state acquisition unit 361 acquires the current position, direction, speed, and positioning status of the vehicle 400. The future prediction unit 362 predicts the position, direction, and speed of the vehicle 4001 and subsequent vehicles at future times. For example, as illustrated in Fig. 20 , the future prediction unit 362 predicts expected positions B1 to B5 of the vehicle 400 at future times t1 to t5 based on information about the position B0, direction, and speed of the vehicle 400 at the current time t0.

[0109] In step S108, future positional relationship estimation unit 371 compares the blind spot position at the estimated position of unmanned aerial vehicle 100 with the estimated position of vehicle 400 at each future time to determine whether vehicle 400 will be hidden in a blind spot. For example, as shown in FIG. 21 , future positional relationship estimation unit 371 compares the blind spot position at estimated positions A1 to A3 of unmanned aerial vehicle 100 at future times t1 to t3 (e.g., elliptical zone 91 at time t3) with estimated positions B1 to B5 of vehicle 400 at future times t1 to t5. As a result of this comparison, future positional relationship estimation unit 371 determines that vehicle 400 will enter the blind spot position (elliptical zone 91) at estimated position B3 at time t3.

[0110] In step S109, when future positional relationship estimation unit 371 determines that vehicle 400 will enter a blind spot, command generation unit 381 generates time-series target values ​​for the position, orientation, and time of unmanned aerial vehicle 100 that will prevent vehicle 400 from entering the blind spot, and generates a movement plan for unmanned aerial vehicle 100 based on the time-series target values. For example, as shown in FIG. 21 , under the constraints of pre-created flight path 81, when it is determined that vehicle 400 will enter blind spot 91 at estimated position A3 of unmanned aerial vehicle 100, command generation unit 381 changes flight path 81 of unmanned aerial vehicle 100 so that vehicle 400 can be tracked and photographed. For example, as shown in FIG. 22 , command generation unit 381 changes the route between estimated positions A2 and A3 on original flight path 81 to a new flight path that takes a shortcut, as indicated by arrow C23 in the figure, and sends this to unmanned aerial vehicle 100 as a control command. As a result, for example, vehicle 400 at positions B3 and B4 can be photographed from future times t3 to t4 without being hidden in a blind spot. Alternatively, for example, command generator 381 generates a new flight path at a higher altitude instead of route 82 from times t2 to t4, which is indicated by a dashed line in flight path 81, as shown in FIG. 23 , and commands unmanned aircraft 100 (its control device 200) to follow this new flight path. This allows unmanned aircraft 100 at higher altitudes to photograph vehicle 400 without being obstructed by obstacle 20. Note that, in addition to the examples shown in FIGS. 22 and 23 , flight control such as changing flight path 81 to move away from or approach course 10 or increasing the speed of travel to the target position can also be used to prevent vehicle 400 from being hidden in a blind spot.

[0111] 22 and 23 have been described as cases in which a flight plan for photographing one vehicle 400 is implemented by one unmanned aerial vehicle 100. Below, with reference to Figures 24 and 25, two types of cases in which the flight plan cannot be implemented by one unmanned aerial vehicle 100 will be described.

[0112] 24 , an example of a case will be described in which the command generation unit 381 determines that the flight plan (target values ​​for time-series position, orientation, and time) cannot be realized by the second unmanned aerial vehicle 100-1 for reasons such as insufficient travel speed performance of the first unmanned aerial vehicle 100-1. In this example, the command generation unit 381 determines whether the second unmanned aerial vehicle 100-1, which has been flying according to the flight plan and photographing the vehicle 400 up until now, will find it difficult (or impossible) to fly according to the flight plan from a certain future time onwards, and if it determines that it is difficult (or impossible), it transmits a tracking standby command including the flight plan for the future time onwards to the second unmanned aerial vehicle 100-2, which is located at another position around the course 10, and hands over the task of tracking and photographing the vehicle 400 (the task of flying and photographing according to that flight plan) to the second unmanned aerial vehicle 100-2. For example, in the example of FIG. 24 , first unmanned aerial vehicle 100-1 reaches positions A0 to A2 according to the flight plan between times t0 and t2 and photographs vehicle 400 at positions B0 to B2, respectively. However, suppose that first unmanned aerial vehicle 100-1 has difficulty flying according to the flight plan at a subsequent time t3. In this case, server 300 issues a tracking standby command to second unmanned aerial vehicle 100-2, which is flying at a different position around course 10, at an appropriate timing before time t3. The command includes a flight plan (time-series target values ​​for position, orientation, and time) for photographing vehicle 400 after time t3 (e.g., between times t3 and t4). In response to this command, second unmanned aerial vehicle 100-2 takes over the tracking and photographing tasks from times t3 to t4. That is, second unmanned aerial vehicle 100-2 reaches target positions D3 and D4 at times t3 and t4, respectively, and, in the target orientation, photographs vehicle 400 without being obstructed by obstacles. In the example shown in Figure 24, during the above process, the next tracking standby command is given to first unmanned aerial vehicle 100-1, and first unmanned aerial vehicle 100-1 moves to target position A5 by the subsequent time t5, and then resumes the task of tracking and photographing vehicle 400 from time t5.

[0113] Referring to FIG. 25 , an example of a case will be described in which command generation unit 381 determines that vehicle 400 is in tunnel 92, which cannot be photographed by unmanned aerial vehicle 100. In this example, the photographing task is assigned to vehicle 400 so that onboard camera 471 is used within tunnel 92. In the example shown in FIG. 25 , unmanned aerial vehicle 100 photographs vehicle 400 at positions A0 to A2, respectively, from time t0 to t2, at positions B0 to B2. Then, prior to time t3 when vehicle 400 is expected to enter tunnel 92, a standby command is issued to onboard camera 471 of vehicle 400, including a command requesting the onboard camera 471 to perform the photographing task until time t4 when vehicle 400 is expected to exit tunnel 92. As a result, while vehicle 400 is traveling within tunnel 92, onboard camera 471 photographs the race. Then, at time t4 when vehicle 400 exits the tunnel, the photographing task is handed over from onboard camera 471 to unmanned aerial vehicle 100 again. Prior to time t4, a tracking standby command including a flight plan (target values ​​for position, orientation, and speed in time series) from time t4 onwards is sent to unmanned aerial vehicle 100, and unmanned aerial vehicle 100 waits until time t4, and then resumes the task of tracking and photographing vehicle 400 from time t4.

[0114] Next, control for a case where there are multiple vehicles 400 to be photographed simultaneously will be described with reference to Figures 26 to 29. When there are multiple vehicles 400 to be photographed simultaneously within course 10 and all of them cannot be accommodated within the field of view of first unmanned aerial vehicle 100-1, or when some of the multiple vehicles 400 are hidden in a blind spot as viewed from first unmanned aerial vehicle 100-1, command generation unit 381 transmits a tracking standby command to second unmanned aerial vehicle 100-2 flying at a different position around course 10, and hands over the task of tracking and photographing at least some of the multiple vehicles 400. In one specific example shown in Figures 26 to 29, two vehicles 400-1, 400-2 traveling on course 10 are photographed by two unmanned aerial vehicles 100-1, 100-2.

[0115] In the example shown in FIGS. 26 to 29, at time t1 shown in FIG. 26, two vehicles 400-1 and 400-2 are located at positions A1 and B1, respectively. Vehicle 400-1 at position A1 is hidden in a blind spot behind obstacle 20 when viewed from the position of unmanned aerial vehicle 100-1. Meanwhile, vehicle 400-1 at position A1 and vehicle 400-2 at position B1 are captured within the angle of view 90 of the camera of unmanned aerial vehicle 100-2. Therefore, command generation unit 381 sends a tracking standby command to unmanned aerial vehicle 100-2, and hands over the task of tracking and photographing the two vehicles 400-1 and 400-2. As a result, unmanned aerial vehicle 100-2 tracks and photographs vehicles 400-1 and 400-2.

[0116] 27 , the two vehicles 400-1 and 400-2 are located at positions A2 and B2, respectively. Vehicle 400-1 at position A2 is captured within angle of view 92 of the camera of unmanned aerial vehicle 100-1. Vehicle 400-2 at position B2 is hidden in a blind spot when viewed from the position of unmanned aerial vehicle 100-1, but is captured within angle of view 90 of the camera of unmanned aerial vehicle 100-2. Therefore, command generation unit 381 transmits a tracking standby command to unmanned aerial vehicle 100-1, and hands over the task of tracking and photographing vehicle 400-1 to unmanned aerial vehicle 100-1.

[0117] At a subsequent time t3 shown in Figure 28, two vehicles 400-1 and 400-2 are located at positions A3 and B3, respectively. Vehicle 400-1 at position A3 is still captured within angle of view 92 of the camera of unmanned aerial vehicle 100-1. Vehicle 400-2 at position B3 is still captured within angle of view 90 of the camera of unmanned aerial vehicle 100-2. Therefore, command generation unit 381 continues from the previous time t2, performing control so that unmanned aerial vehicles 100-1 and 100-2 track and photograph vehicles 400-1 and 400-2, respectively.

[0118] At a subsequent time t4 shown in FIG. 29 , the two vehicles 400-1 and 400-2 are located at positions A4 and B4, respectively. Vehicle 400-1 at position A4 is still captured within the angle of view 90 of the camera of unmanned aerial vehicle 100-2. On the other hand, vehicle 400-2 at position B4 is hidden in the blind spot of the camera of unmanned aerial vehicle 100-1, but is captured within the angle of view 90 of the camera of unmanned aerial vehicle 100-2. Therefore, command generation unit 381 sends a tracking standby command to unmanned aerial vehicle 100-2, and hands over the task of tracking and photographing both vehicles 400-1 and 400-2. As a result, unmanned aerial vehicle 100-2 tracks and photographs vehicles 400-1 and 400-2.

[0119] FIG. 30 is a flowchart showing the determination process of the command generating unit 381.

[0120] In step S201, the command generation unit 381 generates target values ​​for the time series of position, orientation, and time of the photographing camera 111 so that the vehicle 400 is not hidden in a blind spot, and generates a flight plan using the time series target values.

[0121] In step S202, the command generation unit 381 determines whether the generated flight plan can be realized by a single unmanned aerial vehicle 100. In doing so, it determines whether it would be difficult (or impossible) for a single unmanned aerial vehicle 100 to follow the flight plan even if any of the possible changes to the flight plan are made, such as shortcutting the flight path, increasing the altitude, moving away from course 10, or changing the speed. If the generated flight plan can be realized by a single unmanned aerial vehicle 100 (Yes in step S202), the process proceeds to step S203.

[0122] In step S203, the command generation unit 381 determines whether one unmanned aerial vehicle 100 can follow the flight plan without changing the flight plan. If one unmanned aerial vehicle 100 can follow the plan (Yes in step S203), the processing proceeds to step S208, and the command display unit 382 transmits the control command generated by the command generation unit 381 to the control device 200 of that one unmanned aerial vehicle 100, and displays an image indicating the content of the control command on the control device 200.

[0123] On the other hand, if a change in the flight plan is required to allow one unmanned aerial vehicle 100 to follow the flight plan in step S203 (No in step S203), in step S204, the command generation unit 381 changes the flight plan according to the control target values ​​included in the flight plan. For example, the command generation unit 381 changes the flight path within the geofence to generate a shortcut path, increase the flight altitude, move the flight path away from course 10, or move closer to course 10. In addition, for example, the command generation unit 381 may change the flight speed to increase the flight speed to the limit of the aircraft performance. After this, the process proceeds to step S208, and the command display unit 382 transmits the command generated by the command generation unit 381 to the control device 200 and causes the control device 200 to display an image of the control content.

[0124] On the other hand, if the flight plan (target values ​​for position, orientation, and time) cannot be realized by a single unmanned aerial vehicle 100 in step S202 (No in step S202), then in step S205 the command generation unit 381 determines whether or not there is another unmanned aerial vehicle 100 in the vicinity of the course 10 that can track and photograph the vehicle 400 to be photographed. If there is another unmanned aerial vehicle 100 that can be tracked and photographed (Yes in step S205), the process proceeds to step S206.

[0125] In step S206, the command generation unit 381 transmits a tracking standby command to the other unmanned aerial vehicles 100 flying at different positions around the course 10, and hands over the task of tracking and photographing the vehicle 400. After this, the process proceeds to step S208, and the command display unit 382 transmits the control command generated by the command generation unit 381 to the control device 200 of the other unmanned aerial vehicles 100, and also displays an image of the control content.

[0126] On the other hand, if there is no other unmanned aerial vehicle 100 that can track and photograph the vehicle 400 in step S205 (No in step S205), the processing proceeds to step S207.

[0127] In step S207, the command generator 381 makes a more significant change to the flight plan (time series of position, orientation, and time target values), after which the process repeats step S201.

[0128] FIG. 31 is a flowchart showing the determination process of the camera control command unit when an abnormal state is detected.

[0129] In step S301, the command generation unit 381 executes the tracking and photographing task and determines whether or not information indicating an abnormal condition has been acquired from the control device 200 of the unmanned aerial vehicle 100. If information indicating an abnormal condition has been acquired (Yes in step S301), the process proceeds to step S302.

[0130] In step S302, command generation unit 381 determines whether the abnormal condition is an abnormality in control communication between pilot control device 200 and unmanned aerial vehicle 200. If the abnormal condition is an abnormality in control communication (Yes in step S302), processing proceeds to step S303.

[0131] In step S303, the command generation unit 381 determines whether there is another unmanned aerial vehicle 100 to which the tracking and photographing tasks can be handed over, other than the unmanned aerial vehicle 100 in which the control communication abnormality occurred. If such another unmanned aerial vehicle 100 exists (Yes in step S303), the process proceeds to step S304.

[0132] In step S304, command generation unit 381 transmits a tracking standby command to the other unmanned aerial vehicle 100, hands over the tracking and photographing tasks, and thereafter causes the other unmanned aerial vehicle 100 to track and photograph vehicle 400. Then, the process proceeds to step S305.

[0133] In step S305, the command display unit 382 sends the control command generated by the command generation unit 381 to the control device 200 of the other unmanned aerial vehicle 100, and displays an image of the control content.

[0134] On the other hand, if there is no other unmanned aircraft 100 other than the unmanned aircraft that experienced the control communication abnormality in step S303 that can take on the tracking and photographing tasks (No in step S303), in step S306, the command generation unit 381 generates a command to increase the flight altitude of the unmanned aircraft that experienced the control communication abnormality and transmits it to the control device 200. In this way, the command generation unit 381 attempts to restore communication by increasing the altitude of the unmanned aircraft to an altitude at which communication is restored. After this, the processing proceeds to step S307.

[0135] In step S307, the command generation unit 381 determines whether normal control communication has been restored. If control communication has been restored (Yes in step S307), the process proceeds to step S305. If normal control communication has not been restored (No in step S307), the process proceeds to step S308.

[0136] In step S308, the command generation unit 381 generates and transmits a command to cause the unmanned aerial vehicle 100 experiencing the control communication abnormality to perform an evacuation flight (for example, hovering, returning, or soft landing on the spot). After this, the process proceeds to step S305.

[0137] On the other hand, if the abnormal state in step S302 is not a control communication abnormality (No in step S302), in step S309, the command generation unit 381 compares the positioning information (e.g., latitude, longitude, and altitude) of the vehicle (subject) 400 with the position coordinates (e.g., latitude, longitude, and altitude) of the course 10, and determines whether there is a difference between the two that can be considered abnormal. If the difference between the two is abnormal (Yes in step S309), the command generation unit 381 determines that the positioning accuracy of the vehicle 400 has deteriorated to an abnormal level, and the process proceeds to step S310.

[0138] In step S310, the command generation unit 381 uses the positioning information (especially latitude and longitude) transmitted from the vehicle 400 as reference information to set a target value for the flight position within a pre-established geofence (permitted flight area). In this case, the command generation unit 381 sets the target value for the flight position by using the altitude information for the course 100 acquired in advance, instead of the altitude information included in the positioning information from the vehicle 400, and regarding this information as the altitude information for the vehicle 400. The set target value is used in a control command to control the flight of the unmanned aerial vehicle 100. After this, the process proceeds to step S305. Here, if the position measurement unit 461 of the vehicle 400 and the camera position acquisition unit 7131 of the unmanned aerial vehicle 100 use different positioning methods, the unmanned aerial vehicle can be safely flown even if the positioning accuracy of the vehicle 400 deteriorates. Even if both the vehicle 400 and the unmanned aerial vehicle 100 use satellite signals such as GPS, if they use signals from different satellites, the effect of flying the unmanned aerial vehicle safely can be achieved even if the positioning accuracy of the vehicle 400 deteriorates.

[0139] On the other hand, in step S309, when the above difference is not at an abnormal level (No in step S309), the command generation unit 381 determines that the positioning accuracy of the unmanned aircraft 100 has deteriorated to an abnormal level, and generates and transmits a command to cause the unmanned aircraft 100 to perform an evacuation flight (for example, hovering, returning, or soft landing on the spot). Thus, the command generation unit 381 examines the reception environment of satellite signals at the site (for example, whether a sufficient number of satellite signals can be received) and determines whether the unmanned aircraft 100 can fly. After that, the process proceeds to step S305.

[0140] On the other hand, when the abnormal state information is not acquired in step S301 (No in step S301), the process proceeds to step S305.

[0141] As described above, in the present embodiment, by the above processing, a subject moving on a predetermined course can be more appropriately photographed using the moving imaging device.

[0142] Note that events including race interruption, penalty, restart, goal, and course out may be determined based on two or more triggers of the subject's video and flag information (such as the bell in the final lap). When using satellite signals from the artificial satellite 500 for the control communication between the above-described unmanned aircraft 100 and the control device 200 or for positioning the unmanned aircraft 100 or the subject, it is desirable to consider whether sufficient number and intensity of satellite signals can be received at the planned flight position and time of the unmanned aircraft 100 so as to prevent deterioration of the satellite signal reception state as shown in FIG. 31 in advance and determine whether flight is possible. However, even if the satellite signal reception state deteriorates, the unmanned aircraft 100 can be controlled more safely by the determination process shown in FIG. 31.

[0143] In the above embodiment, the control of an unmanned aerial vehicle photographing a car race is described. However, this is merely an example for illustrative purposes and is not intended to limit the scope of the present invention to the above embodiment. The present invention can be embodied in various forms different from the above embodiment. The subject in the present invention is not limited to a car, but may be a person, animal, machine, natural object, or other object moving within a course. Furthermore, the course does not have to be a long, narrow area such as a road where the subject moves essentially in the longitudinal direction. For example, the course may be a vast area such as a yacht race area, a three-dimensional area such as an airplane race airspace, or an area where the subject can move freely in any direction, such as a soccer field or a theater stage. Therefore, for example, the course in the present invention includes courses in racing competitions such as automobiles, motorcycles, bicycles, skiing, marathons, triathlons, yachts, and airplanes. The course may be predetermined before filming, or may be set as needed or in real time during filming. Furthermore, the course, subject, and moving subject are not limited to real objects existing in the real world, but may also be virtual entities existing in a virtual world created by a computer.

[0144] 10: Course 20: Obstacle 90, 92: Shooting angle of view 100: Unmanned aerial vehicle (mobile shooting device) 123: Flight control unit 200: Control device 300: Server 310: Course information acquisition unit 321: Geofence setting unit 322: Flight path setting unit 350: Blind spot estimation unit 400: Vehicle 500: Satellite 600: Reference position communication station 710: Mobile camera (mobile shooting device) 800: Communication network 900: Display device

Claims

1. A system for controlling a mobile photographing device having a camera for photographing a subject moving within a course while moving, comprising: a course information acquisition unit that acquires course information including information regarding the position of the course; an obstacle information acquisition unit that acquires obstacle information including information regarding the positions of obstacles present around the course; a subject information acquisition unit that acquires subject information including information regarding the position of the subject; and a mobile control unit that controls the mobile photographing device based on the course information, the obstacle information, and the subject information so as to avoid the subject being hidden in a blind spot behind the obstacle as viewed from the camera of the mobile photographing device, or so as to avoid the occurrence of a course blind spot, which is a blind spot that hides at least a part of the course within the shooting angle of the camera.

2. A system as described in claim 1, further comprising a blind spot determination unit which determines whether or not there is a blind spot on the course at the current time, or the position of the blind spot which is at least a part of the blind spot on the course, based on the course information and the obstacle information, or a blind spot estimation unit which estimates whether or not there is a blind spot on the course at a future time, or the position of the blind spot, and the mobile control unit controls the mobile photographing device based on the determination result of the blind spot determination unit, or the estimation result of the blind spot estimation unit.

3. A system as described in claim 2, wherein the blind spot estimation unit predicts the expected position of the mobile camera device at the future time, and estimates the presence or absence of a blind spot on the course or the position of the blind spot at the future time based on the expected position, the course information, and the obstacle information.

4. A system as described in claim 2, wherein the blind spot estimation unit obtains a three-dimensional model representing the three-dimensional positional relationship between the course and the obstacle, or generates the three-dimensional model based on the course information and the obstacle information, and estimates the presence or absence of a blind spot on the course or the position of the blind spot at the future time based on the three-dimensional model.

5. A system as described in claim 2, wherein the blind spot determination unit determines the presence or absence of a blind spot on the course or the location of the blind spot at the current time based on the course information, the obstacle information, and information relating to the position of the mobile imaging device or the camera at the current time.

6. A system as described in claim 2, comprising: a subject position prediction unit that predicts the position of the subject at the future time; and a judgment unit that uses the position of the course blind spot at the future time estimated by the blind spot estimation unit and the position of the subject at the future time predicted by the subject position prediction unit to judge whether or not at least a portion of the subject will be in the blind spot at the future time.

7. A system as described in claim 2, wherein the movement control unit determines whether or not at least a portion of the subject is in the blind spot by using the position of the course blind spot at the current time determined by the blind spot determination unit and the position of the subject at the current time acquired by the subject information acquisition unit.

8. A system as described in claim 1, wherein the mobile control unit generates a target value for the position of the mobile camera and a target value for the orientation or zoom amount of the camera to avoid the subject being hidden in the blind spot at a future time based on the estimation result by the blind spot estimation unit, and controls the mobile camera based on the target values.

9. A system as described in claim 1, wherein, when the subject is equipped with a positioning device that measures information regarding the subject's position, or when there is a position information providing device that provides information regarding the subject's position to the system, the subject information acquisition unit acquires information regarding the subject's position or speed from the subject's positioning device or the position information providing device.

10. A system according to claim 1, further comprising a movement prediction unit for predicting an expected position of the subject within the course at a future time.

11. A system as described in claim 10, wherein the movement prediction unit predicts the likely position of the subject at the future time based on at least one of the subject's current movement speed, statistical information regarding the subject's past movements, and pre-recorded reference information regarding the subject's movements.

12. A system as described in claim 1, comprising a geofence generation unit that generates a geofence that defines an area within which the mobile camera device is permitted to move based on the obstacle information so as to avoid interference between the mobile camera device and the obstacle.

13. A system as described in claim 12, wherein the geofence generation unit generates the geofence based on the course information and the obstacle information so as to avoid interference between the mobile camera device and the obstacle, and also to avoid interference between the mobile camera device and the subject within the course.

14. A system as described in claim 12, comprising a movement path generation unit that sets a movement path for the mobile photographing device within the range of the geofence generated by the geofence generation unit and so as to avoid at least a portion of the subject within the course entering the blind spot.

15. A system as described in claim 8, wherein, when a first and a second mobile camera are present at different positions around the course, the movement control unit determines whether it is difficult or impossible for the first mobile camera to track the subject in accordance with the target value, and if it determines that it is difficult or impossible, hands over the task of tracking and photographing the subject in accordance with the target value to the second mobile camera.

16. A system as described in claim 1, wherein, when first and second mobile camera devices are present at different positions around the course and a plurality of subjects are present within the course, the mobile control unit determines whether it is impossible to capture at least some of the plurality of subjects within the shooting angle of view of the first mobile camera device, or whether at least some of the plurality of subjects are hidden in the blind spot as seen by the first mobile camera device, and if it determines that it is impossible or that at least some of the plurality of subjects are hidden, the system hands over the shooting task of tracking and photographing at least some of the plurality of subjects to the second mobile camera device.

17. A system as described in claim 1, comprising a judgment unit that acquires positioning information indicating the three-dimensional position of the subject from a positioning device that measures the three-dimensional position of the subject, compares the positioning information with course coordinates that indicate the three-dimensional position of the course acquired in advance, and judges whether or not the positioning accuracy of the positioning device has deteriorated based on the result of the comparison.

18. A system as described in claim 17, wherein the movement control unit uses information on the altitude of the course indicated by the coordinate information to control the mobile photographing device, instead of the altitude of the subject indicated by the positioning information, when it is determined that the positioning accuracy of the subject has deteriorated.

19. A system as described in claim 17, wherein the movement control unit controls the movement of the mobile photographing device using a previously set permitted movement area for the mobile photographing device, instead of or in addition to the positioning information, when it is determined that the positioning accuracy has deteriorated.

20. A system as described in claim 1, wherein, when the first and second mobile photographing devices are located at different positions around the course, the first mobile photographing device has a first control device and performs a first control communication with the first control device, and the second mobile photographing device has a second control device and performs a second control communication with the second control device, the mobile control unit determines whether an abnormality has occurred in the first control communication between the first mobile photographing device and the first control device, and when it determines that an abnormality has occurred, the system hands over the photographing task of tracking and photographing the subject to the second mobile photographing device.

21. A system for controlling a mobile photography device having a camera for photographing a subject moving within a course while moving, the system having a computer, the computer comprising an arithmetic processing device, a storage device for storing a computer program that can be executed by the arithmetic processing device, and a communication device for communicating with the mobile photography device, the arithmetic processing device reading and executing the computer program from the storage device to: acquire course information including information regarding the position of the course, acquire obstacle information including information regarding the positions of obstacles present around the course, acquire subject information including information regarding the position of the subject, and control the mobile photography device via the communication device based on the course information, obstacle information, and subject information so as to avoid the subject being hidden in a blind spot behind the obstacle as viewed from the camera of the mobile photography device, or so as to avoid the occurrence of a course blind spot, which is a blind spot that hides at least a part of the course within the shooting angle of the camera.

22. A method for controlling a mobile photographing device that utilizes a system for controlling a mobile photographing device having a camera for photographing a subject moving within a course while moving, wherein the system acquires course information including information regarding the position of the course, acquires obstacle information including information regarding the positions of obstacles present around the course, acquires subject information including information regarding the position of the subject, and controls the mobile photographing device via the communication device based on the course information, obstacle information, and subject information so as to avoid the subject being hidden in a blind spot behind the obstacle as viewed from the camera of the mobile photographing device, or so as to avoid the occurrence of a course blind spot, which is a blind spot that hides at least a part of the course within the shooting angle of the camera.

Images