Information processing device, information processing method, and information processing system

Abstract

The present technology relates to an information processing device, an information processing method, and an information processing system that make it possible for an autonomous flying body to execute a task under appropriate weather conditions. This information processing device is provided with a planning unit that creates an execution plan for the autonomous flying body to execute the task, the execution plan including a flight path of the autonomous flying body, on the basis of an estimation result of the weather of a flight region in which the autonomous flying body flies and a weather condition suitable for the task executed by the autonomous flying body. The present technology can be applied to, for example, a drone controller.

Description

Information processing device, information processing method, and information processing system

[0001] The present technology relates to an information processing device, an information processing method, and an information processing system, and in particular to an information processing device, an information processing method, and an information processing system that enable an autonomous air vehicle to perform a task under appropriate weather conditions.

[0002] In recent years, photogrammetry using drones has become widespread. In photogrammetry using drones, for example, three-dimensional data showing the three-dimensional shape of terrain, structures, and the like is generated based on photographs (images) taken from the sky by the drone. Therefore, the accuracy of the three-dimensional data is affected by the condition of the photograph. For example, if photography is performed on a sunny day, the subject on the ground may reflect sunlight, causing the photograph to appear overexposed, which may reduce the accuracy of the three-dimensional data. Therefore, it is desirable to take photographs for surveying on cloudy days.

[0003] Meanwhile, it has been proposed that an autonomous flying vehicle consisting of a High Altitude Platform Station (HAPS) acquires cloud cover information and predicts cloud cover using measuring instruments, and determines a flight path so that the autonomous flying vehicle flies in an area of ​​the stratosphere where the predicted cloud cover is lower than a stratospheric cloud cover threshold, and flies in an area of ​​the troposphere where the predicted cloud cover is lower than a tropospheric cloud cover threshold (see, for example, Patent Document 1). Therefore, clear skies are a preferable weather condition for a HAPS that provides base stations for mobile terminals.

[0004] Patent No. 7076659

[0005] As such, the weather conditions suitable for performing a task vary depending on the task performed by an autonomous air vehicle such as a drone or HAPS.

[0006] The present technology was developed in light of these circumstances, and enables autonomous aerial vehicles to perform tasks in suitable weather conditions.

[0007] An information processing device according to a first aspect of the present technology includes a planning unit that creates an execution plan for the autonomous aircraft to execute a task, including a flight path for the autonomous aircraft, based on predicted weather conditions in a flight target area in which the autonomous aircraft will fly and weather conditions suitable for the task to be executed by the autonomous aircraft.

[0008] In the information processing method of the first aspect of the present technology, an information processing device creates an execution plan for the autonomous aircraft to perform a task, including a flight path for the autonomous aircraft, based on predicted weather conditions in a flight target area in which the autonomous aircraft will fly and weather conditions suitable for the task to be performed by the autonomous aircraft.

[0009] An information processing system according to a second aspect of the present technology includes a weather prediction unit that predicts the weather in a target flight area in which the autonomous air vehicle flies based on weather forecast information for the area, a planning unit that creates an execution plan for the autonomous air vehicle to execute the task, the execution plan including the flight path of the autonomous air vehicle based on the predicted weather in the target flight area and weather conditions suitable for the task to be executed by the autonomous air vehicle, and an operation control unit that causes the autonomous air vehicle to execute the task in accordance with the execution plan, and the weather prediction unit, the planning unit, and the operation control unit are each located in one of two or more information processing devices.

[0010] In a first aspect of the present technology, an execution plan is created for the autonomous aircraft to execute a task, including a flight path for the autonomous aircraft, based on predicted weather conditions in a flight target area in which the autonomous aircraft will fly and weather conditions suitable for the task to be executed by the autonomous aircraft.

[0011] In a second aspect of the present technology, the weather in a target flight area in which an autonomous air vehicle is to fly is predicted based on weather forecast information for the vicinity of the target flight area, and an execution plan is created for the autonomous air vehicle to execute the task, including the flight path of the autonomous air vehicle, based on the predicted weather in the target flight area and weather conditions suitable for the task to be executed by the autonomous air vehicle, and the task is executed by the autonomous air vehicle in accordance with the execution plan.

[0012] 1 is a block diagram showing an embodiment of an information processing system to which the present technology is applied. FIG. 1 is a block diagram showing an example of a functional configuration of a drone. FIG. 1 is a block diagram showing an example of an external configuration of a drone. FIG. 1 is a block diagram showing an example of a functional configuration of a controller. FIG. 1 is a block diagram showing an example of a functional configuration of a weather observation device. FIG. 1 is a block diagram showing an example of a functional configuration of a server. FIG. 2 is a flowchart for explaining an execution plan creation process. FIG. 3 is a diagram showing an example of a flight path. FIG. 4 is a diagram showing a method for correcting an execution plan. FIG. 5 is a diagram showing an example of an execution plan selection screen. FIG. 6 is a flowchart for explaining a task execution control process. FIG. 7 is a diagram showing an example of a method for predicting weather in a target flight area. FIG. 8 is a diagram showing an example of a method for predicting weather in a target flight area. FIG. 9 is a diagram showing an example of a method for correcting an execution plan. FIG. 10 is a diagram showing an example of a method for correcting an execution plan. FIG. 11 is a diagram showing an example of a method for correcting an execution plan. FIG. 12 is a diagram showing an example of a method for correcting an execution plan. FIG. 13 is a diagram showing an example of an execution status display screen in normal times. FIG. 14 is a diagram showing an example of an execution status display screen in the case where a correction proposal for an execution plan is presented. FIG. 15 is a block diagram showing an example of the configuration of a computer.

[0013] Hereinafter, embodiments of the present technology will be described. The description will be made in the following order: 1. Embodiment 2. Modification 3. Other

[0014] <<1. Embodiment>> An embodiment of the present technology will be described with reference to FIGS. 1 to 21 .

[0015] <Configuration Example of Information Processing System 1> First, with reference to FIG. 1, a configuration example of an information processing system 1 to which the present technology is applied will be described.

[0016] The information processing system 1 is a system that, for example, includes a flight path of the drone 11, creates an execution plan for the drone 11 to execute a specified task, and causes the drone 11 to execute the task according to the execution plan.

[0017] Here, the content of the task is not particularly limited as long as it can be performed autonomously by the drone 11. The following mainly describes an example in which the drone 11 performs a task of taking photographs for photogrammetry.

[0018] The information processing system 1 includes a drone 11, a controller 12, a weather observation device 13A, a weather observation device 13B, and a server 14.

[0019] The drone 11, the controller 12, the weather observation device 13A, the weather observation device 13B, and the server 14 are connected to each other via a network 21 and can communicate with each other. The network 21 includes various networks such as the Internet and a mobile phone network, base stations, access points, etc. The drone 11, the controller 12, and the weather observation device 13A can also communicate directly with each other via wireless or wired communication without going through the network 21.

[0020] In Figure 1, the number of drones 11, controllers 12, weather observation devices 13A, weather observation devices 13B, and servers 14 is shown as one each, but the number is not particularly limited, and multiple of each may be provided.

[0021] Hereinafter, when there is no need to distinguish between the meteorological observation device 13A and the meteorological observation device 13B, they will be simply referred to as the meteorological observation device 13.

[0022] Hereinafter, when each device communicates via the network 21, the description of the network 21 may be omitted. For example, when the drone 11 communicates with the server 14 via the network 21, it may be described as the drone 11 communicating with the server 14.

[0023] The drone 11 flies and performs various tasks autonomously or under remote control by the controller 12 or the server 14. The drone 11 generates, transmits to the outside, and receives from the outside information required to create an execution plan for executing various tasks.

[0024] For example, the drone 11 receives, from the controller 12, operation commands or control parameters for remotely controlling the drone 11. The drone 11 performs various operations by remote control based on the received operation commands or control parameters.

[0025] The drone 11 receives execution plan information indicating an execution plan for executing various tasks from the controller 12 and the server 14. The drone 11 flies autonomously and executes various tasks in accordance with the execution plan based on the received execution plan information.

[0026] The drone 11 receives task information related to the tasks to be performed by the drone 11 from the controller 12. The drone 11 receives weather forecast information indicating the results of forecasting the weather near the area in which the drone 11 will fly to perform various tasks (hereinafter referred to as the target flight area) from the server 14. The drone 11 receives weather observation information indicating the results of observing the weather near the target flight area from the weather observation device 13 and the server 14. The drone 11 receives weather prediction information indicating the results of predicting the weather near the target flight area from the controller 12 and the server 14.

[0027] The drone 11 observes the state of the drone 11 and the state around the drone 11. The drone 11 generates state observation information indicating the results of observing the state of the drone 11 and the state around the drone 11, and transmits it to the controller 12 and the server 14.

[0028] The drone 11 predicts the weather near the target flight area based on at least some of the status observation information, externally received weather forecast information, and externally received weather observation information. The drone 11 generates weather prediction information indicating the results of predicting the weather near the target flight area and transmits it to the controller 12 and the server 14.

[0029] The drone 11 creates or corrects an execution plan for executing various tasks based on at least a part of the task information, the status observation information, and the weather prediction information. The drone 11 flies autonomously and executes various tasks in accordance with the created or corrected execution plan.

[0030] The controller 12 is configured, for example, by a dedicated controller or an information processing terminal such as a smartphone, a tablet terminal, or a PC (personal computer).

[0031] The controller 12 is used for remote control of the drone 11. The controller 12 generates information required to create an execution plan for the drone 11, transmits the information to the outside, and receives the information from the outside.

[0032] For example, based on input information input by a user, the controller 12 generates an operation command or control parameters for remotely controlling the drone 11. The controller 12 transmits the operation command or control parameters to the drone 11.

[0033] The controller 12 generates task information related to a task to be executed by the drone 11 based on input information input by the user, etc. The controller 12 transmits the task information to the drone 11 and the server 14.

[0034] The controller 12 receives status observation information indicating the observation results of the status of the drone 11 and its surroundings from the drone 11. The controller 12 receives weather forecast information for the vicinity of the target flight area from the server 14. The controller 12 receives weather observation information for the vicinity of the target flight area from the weather observation device 13 and the server 14. The controller 12 receives weather prediction information for the vicinity of the target flight area from the drone 11 and the server 14.

[0035] The controller 12 predicts the weather near the target flight area based on at least some of the status observation information, weather forecast information, and weather condition information received from outside. The controller 12 generates weather prediction information indicating the predicted results of the weather near the target flight area and transmits it to the drone 11 and the server 14.

[0036] The controller 12 creates or corrects an execution plan for the drone 11 to execute various tasks based on at least some of the task information, the weather prediction information generated by the controller 12, the weather prediction information received from outside, and the status observation information received from outside. The controller 12 generates execution plan information indicating the created or corrected execution plan and transmits it to the drone 11.

[0037] The controller 12 receives execution plan information for the drone 11 from the server 14. The controller 12 corrects the received execution plan information as necessary and transmits it to the drone 11.

[0038] The weather observation device 13 observes the surrounding weather. The weather observation device 13 generates weather observation information indicating the observation results and transmits the information to the drone 11, the controller 12, and the server 14.

[0039] The server 14 externally collects, generates, and externally transmits information necessary for creating an execution plan for the drone 11. The server 14 creates or corrects an execution plan for the drone, and externally transmits execution plan information indicating the created or corrected execution plan.

[0040] For example, the server 14 receives task information related to tasks to be executed by the drone 11 from the controller 12. For example, the server 14 generates weather forecast information for various locations, receives it from an external server or the like, and transmits it to the drone 11 and the controller 12. The server 14 receives weather observation information for various locations from the outside (for example, a weather observation device 13), and transmits it to the drone 11 and the controller 12. The controller 12 receives status observation information from the drone 11 that indicates the observation results of the status of the drone 11 and its surroundings.

[0041] The server 14 predicts the weather in the target flight area based on at least some of the status observation information, weather forecast information, and weather observation information. The server 14 generates weather prediction information indicating the predicted weather in the target flight area and transmits it to the drone 11 and the controller 12.

[0042] The server 14 creates or corrects an execution plan for the drone 11 based on at least a part of the task information, the weather prediction information generated by the server 14 itself, and the weather prediction information received from an external source. The server 14 generates execution plan information indicating the created or corrected execution plan and transmits it to the drone 11 and the controller 12.

[0043] The meteorological elements and meteorological phenomena covered by the above-described weather forecast information and meteorological observation information are not particularly limited as long as they have the potential to affect the execution of tasks by the drone 11. For example, the meteorological elements covered by the weather forecast information and meteorological observation information include one or more of weather, cloud cover, cloud shape, cloud height, visibility, sunshine, air pressure, temperature, humidity, precipitable water, precipitation, snowfall, wind direction, wind speed, atmospheric stability, and the like.

[0044] Furthermore, the content and information sources of the weather forecast information and weather observation information are not particularly limited. For example, in addition to the drone 11, the weather observation device 13, and the server 14, other servers, other drones, weather satellites, weather radar, posts on SNS (social networking services), etc. are possible sources of information. However, it is desirable to use highly reliable information sources.

[0045] <Configuration Example of Drone 11> Next, a configuration example of the drone 11 will be described with reference to FIGS. 2 and 3 .

[0046] FIG. 2 shows an example of the functional configuration of the drone 11.

[0047] The drone 11 includes a sensor unit 51, a photographing unit 52, an information processing unit 53, a power unit 54, a communication unit 55, and a memory unit 56.

[0048] The sensor unit 51 includes various sensors that detect the state of the drone 11 or the state of the surroundings of the drone 11. For example, the sensor unit 51 includes an anemometer, a geomagnetic sensor, an IMU (Inertial Measurement Unit), a GNSS (Global Navigation Satellite System) receiver, etc. Each sensor of the sensor unit 51 supplies sensor data indicating the detection results to the information processing unit 53 and stores the sensor data in the memory unit 56.

[0049] The photographing unit 52 includes various image sensors that photograph the surroundings of the drone 11. For example, the photographing unit 52 includes a camera, a stereo camera, a depth sensor (e.g., a ToF (Time of Flight) sensor), etc. Each image sensor of the photographing unit 52 supplies image data such as photos and videos obtained as a result of photographing to the information processing unit 53 or stores the image data in the storage unit 56.

[0050] The information processing unit 53 is configured by an information processing device (e.g., a microcomputer) including a processor such as a CPU (Central Processing Unit), a memory, etc. The information processing unit 53 controls each part of the drone 11 and executes various processes by executing a predetermined program. The information processing unit 53 includes an observation unit 71, a weather prediction unit 72, a planning unit 73, and an operation control unit 74.

[0051] The observation unit 71 observes the state of the drone 11 and the state of its surroundings based on the sensor data from the sensor unit 51 and the image data from the photographing unit 52. For example, the observation unit 71 observes the weather around the drone 11. The observation unit 71 generates state observation information that indicates the observation results of the state of the drone 11 and the state of its surroundings. The observation unit 71 transmits the state observation information to the controller 12 and the server 14.

[0052] The weather prediction unit 72 predicts the weather near the target flight area based on at least some of the condition observation information from the observation unit 71, the weather observation information from the weather observation device 13, the weather observation information from the server 14, and the weather forecast information from the server 14. The weather prediction unit 72 generates weather prediction information indicating the predicted results of the weather near the target flight area and transmits it to the controller 12 and the server 14.

[0053] The planning unit 73 creates or corrects an execution plan for executing various tasks based on at least some of the task information from the controller 12, the state observation information from the observation unit 71, the weather prediction information from the controller 12, and the weather prediction information from the server 14.

[0054] The operation control unit 74 controls the sensor unit 51, the photographing unit 52, and the power unit 54 based on the execution plan created or corrected by the planning unit 73, operation commands from the controller 12, or control parameters, or execution plan information from the controller 12 or the server 14, thereby controlling the operation of the drone 11.

[0055] The power unit 54 includes a mechanism for flying the drone 11. For example, the power unit 54 includes a propeller, a motor for rotating the propeller, etc. The power unit 54 is driven under the control of the operation control unit 74 to fly the drone 11.

[0056] The communication unit 55 includes various communication devices and performs wireless communication with other devices using a predetermined communication method. For example, the communication unit 55 communicates with the controller 12 and the meteorological observation device 13A, and with the meteorological observation device 13B and the server 14 via the network 21.

[0057] The memory unit 56 includes, for example, a non-volatile memory, and stores various information necessary for processing by the drone 11.

[0058] Figure 3 shows an example of the external configuration of the drone 11 and an example of the installation position of the camera provided in the photographing unit 52.

[0059] A camera 52F for aircraft control is provided in front of the main body 11A of the drone 11. A camera 52L for aircraft control is provided to the left of the main body 11A of the drone 11. A camera 52R for aircraft control is provided to the right of the main body 11A of the drone 11. A camera 52B for aircraft control is provided behind the main body 11A of the drone 11. A camera 52O for surveying photography is provided below the main body 11A of the drone 11.

[0060] <Configuration Example of Controller 12> FIG. 4 shows a functional configuration example of the controller 12. As shown in FIG.

[0061] The controller 12 includes an input unit 101 , an information processing unit 102 , an output unit 103 , and a communication unit 104 .

[0062] The input unit 101 includes various input devices and operation devices, and is used to input various information and operate the controller 12. The input unit 101 supplies input information input using the input devices or operation devices to the information processing unit 102.

[0063] The information processing unit 102 is configured, for example, by an information processing device (for example, a microcomputer) including a processor such as a CPU, a memory, etc. The information processing unit 102 controls each unit of the controller 12 and executes various processes by executing a predetermined program. The information processing unit 102 includes a weather prediction unit 121, a planning unit 122, an output control unit 123, and an operation control unit 124.

[0064] The weather prediction unit 121 predicts the weather near the target flight area based on at least some of the status observation information from the drone 11, the weather observation information from the weather observation device 13, and the weather forecast information and weather observation information from the server 14. The weather prediction unit 121 generates weather prediction information indicating the predicted results of the weather near the target flight area, and transmits this information to the drone 11 via the communication unit 104, or to the server 14 via the communication unit 104 and the network 21.

[0065] Based on the input information, the planner 122 generates task information related to the task to be executed by the drone 11. The planner 122 creates or corrects an execution plan for the drone 11 based on at least some of the task information, status observation information from the drone 11, weather prediction information from the drone 11, weather prediction information from the server 14, and weather prediction information from the weather prediction unit 121. The planner 122 corrects the execution plan information from the server 14 as necessary.

[0066] The output control unit 123 controls the output of various types of information (for example, visual information, auditory information, and tactile information) from the output unit 103.

[0067] The operation control unit 124 performs remote control of the operation of the drone 11.

[0068] For example, the operation control unit 124 generates operation commands or control parameters for remotely controlling the drone 11 based on the input information and state observation information from the drone 11, and transmits them to the drone 11 via the communication unit 104. In this way, the operation of the drone 11 is remotely controlled.

[0069] For example, the operation control unit 124 transmits the execution plan information created or corrected by the planning unit 122 to the drone 11 via the communication unit 104. As a result, the operation of the drone 11 is remotely controlled so that the task is executed in accordance with the execution plan.

[0070] For example, the operation control unit 124 transmits the task information created by the planning unit 122 to the drone 11 via the communication unit 104, or transmits the task information to the server 14 via the communication unit 104 and the network 21. As a result, the operation of the drone 11 is remotely controlled directly or via the server 14 so as to execute the task indicated in the task information.

[0071] The output unit 103 includes various output devices that output visual information, auditory information, or tactile information. For example, the output unit 103 includes a display device, an audio output device, a haptic device, etc. The output unit 103 outputs various types of information under the control of the output control unit 123.

[0072] The communication unit 104 includes various communication devices and performs wireless communication with other devices using a predetermined communication method. For example, the communication unit 104 communicates with the drone 11 and the meteorological observation device 13A, and with the meteorological observation device 13B and the server 14 via the network 21.

[0073] The storage unit 105 includes, for example, a nonvolatile memory, and stores various information necessary for the processing of the controller 12 .

[0074] <Configuration Example of Meteorological Observation Device 13> FIG. 5 shows a configuration example of the functions of the meteorological observation device 13. As shown in FIG.

[0075] The meteorological observation device 13 includes a solar radiation meter 151 , a wind direction and speed meter 152 , a barometer 153 , a thermo-hygrometer 154 , a ceilometer 155 , a rain gauge 156 , an imaging unit 157 , an information processing unit 158 ​​, a communication unit 159 , and a memory unit 160 .

[0076] The solar radiation meter 151 measures the amount of solar radiation and sunlight in the surrounding area, and supplies measurement data indicating the measurement results to the information processing unit 158.

[0077] The anemometer 152 measures the direction and speed of the surrounding wind, and supplies measurement data indicating the measurement results to the information processing unit 158 ​​.

[0078] The barometer 153 measures the ambient air pressure and supplies measurement data indicating the measurement results to the information processing unit 158 ​​.

[0079] The thermo-hygrometer 154 measures the ambient temperature and humidity, and supplies measurement data indicating the measurement results to the information processing unit 158 ​​.

[0080] The ceilometer 155 measures the height of the surrounding cloud base (cloud height) and supplies measurement data indicating the measurement results to the information processing unit 158 ​​.

[0081] The rain gauge 156 measures the amount of precipitation in the surrounding area and supplies measurement data indicating the measurement results to the information processing unit 158.

[0082] The photographing unit 157 is equipped with various cameras, photographs the surroundings, and supplies image data obtained as a result of the photographing to the information processing unit 158 ​​.

[0083] The information processing unit 158 ​​is configured by, for example, an information processing device (for example, a microcomputer) including a processor such as a CPU, a memory, etc. The information processing unit 158 ​​controls each unit of the weather observation device 13 and executes various processes by executing a predetermined program. The information processing unit 158 ​​includes an observation unit 171.

[0084] The observation unit 171 observes the surrounding weather based on the measurement data and image data, and generates meteorological observation information indicating the observation results. The observation unit 171 transmits the meteorological observation information to the drone 11, the controller 12, and the server 14 via the communication unit 159 and, if necessary, via the network 21.

[0085] The communication unit 159 includes various communication devices and communicates with other devices using a predetermined communication method. For example, the communication unit 159 communicates with the drone 11 and the controller 12 via the network 21 or without the network 21, and communicates with the server 14 via the network 21.

[0086] The storage unit 160 includes, for example, a non-volatile memory, and stores various information necessary for the processing of the meteorological observation device 13 .

[0087] The types of weather observation sensors shown in this diagram are just an example, and other types of sensors can be added as needed, or unnecessary sensors can be deleted.

[0088] <Configuration Example of Server 14> FIG. 6 shows a functional configuration example of the server 14. As shown in FIG.

[0089] The server 14 includes an information processing unit 201 , a communication unit 202 , and a storage unit 203 .

[0090] The information processing unit 201 is configured, for example, by an information processing device (for example, a microcomputer) including a processor such as a CPU, a memory, etc. The information processing unit 201 controls each unit of the server 14 and executes various processes by executing a predetermined program. The information processing unit 201 includes an information collection unit 221, a weather prediction unit 222, and a planning unit 223.

[0091] The information collection unit 221 collects information about the weather in various locations, such as weather forecast information, weather observation information, and weather prediction information, from the drone 11, the controller 12, the weather observation device 13, other servers, weather satellites, weather radar, etc., via the network 21 and the communication unit 202. The information collection unit 221 transmits the collected information to the drone 11 and the controller 12 via the communication unit 202 and the network 21.

[0092] The weather prediction unit 222 predicts the weather near the target flight area of ​​the drone 11 based on at least a part of the information collected by the information collection unit 221. The weather prediction unit 222 generates weather prediction information indicating the prediction result and transmits the weather prediction information to the drone 11 and the controller 12 via the communication unit 202 and the network 21.

[0093] The planner 223 creates or corrects an execution plan for causing the drone 11 to execute a task, based on at least one of the status observation information from the drone 11, the weather prediction information from the drone 11, the weather prediction information from the controller 12, and the weather prediction information from the weather prediction unit 222, as well as task information from the controller 12. The planner 223 generates execution plan information indicating the created or corrected execution plan, and transmits it to the drone 11 and the controller 12 via the communication unit 202 and the network 21.

[0094] The communication unit 202 includes various communication devices and communicates with other devices using a predetermined communication method. For example, the communication unit 159 communicates with the drone 11 and the controller 12 via the network 21 or without the network 21, and communicates with the server 14 via the network 21.

[0095] The storage unit 203 includes, for example, a nonvolatile memory, and stores various information necessary for the processing of the server 14 .

[0096] As described above, the creation and correction of the execution plan can be performed by any of the drone 11, the controller 12, and the server 14. Note that, below, an example in which the controller 12 creates and corrects the execution plan will be described.

[0097] <Execution Plan Creation Process> Next, the execution plan creation process executed by the controller 12 will be described with reference to the flowchart of FIG.

[0098] In step S1, the planning unit 122 sets a target flight area.

[0099] For example, the user inputs information about an area to be surveyed (hereinafter referred to as a survey target area) via the input unit 101 .

[0100] In response to this, the input unit 101 supplies the input information to the information processing unit 102. The planning unit 122 sets, for example, an area where the drone 11 needs to fly in order to photograph the survey target area as the flight target area.

[0101] In step S2, the controller 12 acquires setting information.

[0102] For example, the user inputs setting information regarding various setting values, side lap rate, overlap rate, etc. of the imaging unit 52 of the drone 11 via the input unit 101.

[0103] In response to this, the input unit 101 supplies the input information to the information processing unit 102 .

[0104] For example, the information processing unit 53 of the drone 11 may acquire information from the photographing unit 52 (for example, information about the camera and lens) and transfer it to the controller 12. Then, the information processing unit 102 of the controller 13 may use the information acquired from the drone 12 as setting information.

[0105] In step S3, the controller 12 obtains information about the period for executing the task.

[0106] For example, the user inputs information regarding the period for executing a task (photographing) (hereinafter referred to as the task execution period) via the input unit 101. At this time, for example, the user may input only the date and time for starting the task (hereinafter referred to as the task start date and time), or may also input the date and time for ending the task (hereinafter referred to as the task end date and time). Alternatively, for example, the user may input both the task start date and time and the length of the period for executing the task (hereinafter referred to as the task execution time).

[0107] In response to this, the input unit 101 supplies the input information to the information processing unit 102 .

[0108] In step S4, the planner 122 creates a base execution plan. For example, the planner 122 creates the execution plan based on the flight target area, the settings of the image capture unit 52, the side lap rate, the overlap rate, the task execution period, etc. For example, the planner 122 sets a flight path, flight altitude, flight speed, and positions for image capture (hereinafter referred to as image capture positions) that enable the survey target area to be captured at the set side lap rate and overlap rate. Furthermore, the planner 122 sets the takeoff point, final point, and scheduled date and time for arrival at each image capture position of the flight path based on the task execution period.

[0109] FIG. 8 shows an example of a flight target area A1 and a flight route R1 in a base execution plan.

[0110] The target flight area A1 is a rectangular area surrounded by sides A1a to A1d. Note that the target flight area A1 does not necessarily have to be rectangular.

[0111] In the following description, the direction parallel to sides A1a and A1c is referred to as the horizontal direction, and the direction parallel to sides A1b and A1d is referred to as the vertical direction. Hereinafter, side A1a is referred to as the upper side of the target flight area A1, side A1c is referred to as the lower side of the target flight area A1, side A1b is referred to as the right side of the target flight area A1, and side A1d is referred to as the left side of the target flight area A1.

[0112] The flight route R1 starts at takeoff point Ps1 near the upper right corner of the target flight area A1, and alternates between a vertical route from top to bottom and a vertical route from bottom to top, shifting slightly from right to left. After reaching final point Pe1 near the lower left corner of the target flight area A1, flight route R1 connects final point Pe1 and takeoff point Ps1 in a straight line and ends at takeoff point Ps1. Photographing is performed along the vertical route indicated by the solid arrow. Photographing is not performed along the horizontal route indicated by the dotted line or the route from final point Pe1 to takeoff point Ps1. Photographing can also be performed along the dotted line route.

[0113] In step S5, the planner 122 determines whether to correct the execution plan based on the weather in the target flight area.

[0114] For example, the user selects whether or not to correct the execution plan based on the weather in the target flight area, and inputs the selection result via the input unit 101 .

[0115] In response to this, the input unit 101 supplies the input information to the information processing unit 102. If the planning unit 122 determines, based on the information input by the user, that the execution plan should be corrected based on the weather in the target flight area, the process proceeds to step S6.

[0116] In step S6, the weather prediction unit 121 determines whether or not there is usable weather forecast information.

[0117] For example, there are multiple pieces of weather forecast information based on the weather phenomenon to be forecast, the area to be forecast, the period to be forecast, etc. Furthermore, the type and number of available weather forecast information varies depending on the area, season, time period, etc. in which the task is performed.

[0118] Furthermore, for example, the creation of an execution plan may be performed on-site on the day of the task or in advance. Furthermore, the closer to the task execution period, the more types and quantities of weather forecast information for the vicinity of the task execution period become available, and the more accurate the forecast becomes. For example, the weather radar forecast information of the Japan Meteorological Agency can only be obtained up to one hour ahead.

[0119] Furthermore, there is weather forecast information that cannot be obtained unless the user signs a contract, for example.

[0120] In contrast, if the weather prediction unit 121 can obtain weather forecast information for the vicinity of the target flight area for a period including the task execution period, and if the available weather forecast information contains information that can be used to correct the execution plan, it determines that usable weather forecast information exists, and processing proceeds to step S7.

[0121] In step S7, the weather prediction unit 121 acquires usable weather forecast information. That is, the planning unit 122 receives the weather forecast information determined to be usable in the processing of step S6 from the server 14 or the like via the network 21 and the communication unit 104.

[0122] In step S8, the weather prediction unit 121 predicts the weather changes in the target flight area. Specifically, the weather prediction unit 121 predicts the weather changes in the area including the target flight area (hereinafter referred to as the prediction target area) during the task execution period and the periods before and after that (hereinafter referred to as the prediction target period) based on the acquired weather forecast information.

[0123] For example, the weather prediction unit 121 predicts the time series transition of the weather distribution in the prediction target area during the prediction target period. The weather distribution may be, for example, a distribution of weather types such as sunny, cloudy, and rainy, or a distribution of cloud thickness and precipitation amount.

[0124] In step S9, the planner 122 prepares a revised execution plan based on the predicted weather conditions.

[0125] For example, the planner 122 creates a proposed correction to the execution plan using the following three methods.

[0126] Specifically, for example, the planner 122 corrects the execution plan by correcting the task execution period (task start date and time) without changing the flight path. For example, the planner 122 corrects the execution plan by correcting the flight path without changing the task execution period. For example, the planner 122 corrects the execution plan by correcting both the flight path and the task execution period.

[0127] In either case, the planning unit 122 corrects at least one of the flight path and the task execution period so as to increase the proportion of weather conditions suitable for the task (probability of cloudy weather) at the location (photography location) where the drone 11 executes the task, within the scope that allows the execution plan to be completed within the specified time.

[0128] For example, Figures 9A and 9B show examples of predicted weather conditions superimposed on the flight route R1 of Figure 8. In this figure, flight route R1 is divided into flight routes R1a to R1d.

[0129] For example, if the task execution period is 60 minutes long, A in Figure 9 shows the predicted weather for the first half of the task execution period (from the task start time to 30 minutes later). The predicted weather for rectangular area A2, which includes flight route R1a in the first half of flight route R1, is sunny, and the predicted weather for rectangular area A3, which includes flight route R1c in the second half of flight route R1, is cloudy.

[0130] 9B shows the predicted weather for the second half of the task execution period (30 to 60 minutes later). The predicted weather for area A2 has changed to cloudy, and the predicted weather for area A3 has changed to rain.

[0131] In this example, if the task is executed according to flight route R1, it is predicted that photography will be performed in sunny conditions in area A2, and the task will be canceled in area A3 due to rain.

[0132] In response to this, the flight path is corrected, for example, as shown in FIGS.

[0133] Specifically, after moving in a straight line from takeoff point Ps1 to final point Pe1, the flight path is corrected so as to move in the opposite direction along flight path R1.

[0134] As a result, as shown in A of Fig. 10, in the first half of the task execution period, a flight route R2b is set in the cloudy area A3 in the opposite direction to the flight route R1c in B of Fig. 9. Also, as shown in B of Fig. 10, in the second half of the task execution period, a flight route R2c is set in the cloudy area A2 in the opposite direction to the flight route R1a in A of Fig. 9.

[0135] This will ensure that images are taken under cloudy skies for almost the entire task execution period, which is expected to improve the accuracy of the three-dimensional data showing the survey results.

[0136] In step S10, the output unit 103 presents a proposed correction to the execution plan under the control of the output control unit 123. For example, the output unit 103 displays an execution plan selection screen shown in Fig. 11. This selection screen includes execution plan information indicating (the proposed correction to) the execution plan.

[0137] Buttons 301a to 301d are arranged vertically on the left side of this selection screen. Buttons 301a to 301d correspond to execution plans 1 to 4, respectively. The contents of the execution plan corresponding to the button selected with the cursor among buttons 301a to 301d are displayed in a window 302 on the right side.

[0138] A map 311 is located in the center of the window 302. The map 311 includes the target flight area and shows a map of the area in which the task will be executed. Information indicating the currently selected execution plan (Execution Plan 2) is displayed in the upper center of the map 311. The date on which the task will be executed (June 1, 2023) is displayed in the upper right corner of the map 311.

[0139] Below the map 311 are arranged a timeline bar 312 and a playhead 313 for specifying the time at which information is to be displayed within the map 311. Below and in the center of the timeline bar 312, a play button 314 and a stop button 315 are arranged side by side.

[0140] The start time (10:10) of the task execution period is displayed above the left end of the timeline bar 312. The end time (10:30) of the task execution period is displayed above the right end of the timeline bar 312. The specified time (10:14) specified by the playhead 313 is displayed above the playhead 313.

[0141] The flight path in the selected execution plan and the predicted weather for the target flight area at the specified time are superimposed on the map 311. In this example, the cloudy, sunny, and rainy areas within the target flight area at the specified time are displayed in different display modes to show the weather distribution within the target flight area. A legend indicating the display mode of each area is displayed in the lower right corner of the map 311.

[0142] An icon 316 indicating the position of the drone 11 at the specified time is superimposed on the map 311. At the top left edge of the map 311, the weather (cloudy) at the position of the drone 11 indicated by the icon 316 is displayed.

[0143] For example, when the play button 314 is pressed, the playhead 313 shifts from left to right on the timeline bar 312, and the specified time advances. In addition, as the specified time advances, the display of the weather distribution on the map 311 changes, and the icon 316 indicating the position of the drone 11 moves along the flight path.

[0144] On the other hand, when the stop button 315 is pressed, the play button 314 stops, and an icon 316 indicating the weather distribution at the specified time when the play button 314 stopped and the position of the drone 11 is displayed on the map 311. The user can also specify a specified time by directly sliding the play button 314 on the timeline bar 312.

[0145] This allows the user to easily check the flight path for each execution plan, as well as the time series changes in the weather distribution for each execution plan and the time series movements of the drone 11.

[0146] In step S11, the planner 122 saves the selected execution plan.

[0147] Specifically, the user selects a desired execution plan from the options presented in the processing of step S10 via the input unit 101.

[0148] In response to this, the input unit 101 supplies the input information to the information processing unit 102. The planning unit 122 generates execution plan information indicating the execution plan selected by the user, and stores it in the storage unit 105.

[0149] Thereafter, the execution plan creation process ends.

[0150] On the other hand, in step S6, if the weather prediction unit 121 cannot obtain weather forecast information for the vicinity of the target flight area for the period including the task execution period, or if the available weather forecast information does not contain information that can be used to correct the execution plan, it determines that there is no usable weather forecast information, and processing proceeds to step S12.

[0151] Also, if it is determined in step S5 that the execution plan will not be corrected based on the weather in the target flight area, the process proceeds to step S12.

[0152] In step S12, the planning unit 122 saves the base execution plan. Specifically, the planning unit 122 generates execution plan information indicating the base execution plan and stores it in the storage unit 105.

[0153] Thereafter, the execution plan creation process ends.

[0154] <Task Execution Control Processing> Next, the task execution control processing executed by the controller 12 will be described with reference to the flowchart of FIG.

[0155] This process is started, for example, when the start date and time of the execution plan arrives.

[0156] In step S51, the operation control unit 124 reads out an execution plan. That is, the operation control unit 124 reads out, from the storage unit 105, execution plan information for causing the drone 11 to execute the execution plan.

[0157] In step S52, the operation control unit 124 causes the drone 11 to start executing the task. Specifically, the operation control unit 124 transmits the execution plan information to the drone 11 via the communication unit 104.

[0158] In response, the information processing unit 53 of the drone 11 receives the execution plan information via the communication unit 55. The operation control unit 74 starts controlling the imaging unit 52 and the power unit 54 based on the execution plan information. As a result, the drone 11 starts executing the task in accordance with the execution plan.

[0159] In step S53, the planner 122 determines whether or not to correct the execution plan in real time.

[0160] For example, the user sets whether or not to correct the execution plan in real time via the input unit 101 in advance or during the execution of the task.

[0161] If the planning unit 122 determines that the execution plan should be corrected in real time based on the user settings, the process proceeds to step S54.

[0162] In step S54, the controller 12 estimates the transition of the non-cloudy area within the target flight area. For example, the information processing unit 102 receives status observation information indicating the observation results of the drone 11 and the status around the drone 11 from the drone 11 via the communication unit 104. The information processing unit 102 receives weather observation information from the weather observation device 13 via the communication unit 104 and, if necessary, the network 21. The information processing unit 102 receives weather forecast information and weather observation information from the server 14 via the network 21 and the communication unit 104.

[0163] The weather prediction unit 121 predicts the transition of non-cloudy areas within the target flight area based on at least a part of the received information. That is, the weather prediction unit 121 predicts the time series transition of non-cloudy areas within the target flight area that do not satisfy the weather conditions suitable for taking surveying photographs (hereinafter referred to as unsatisfied areas) during the future task execution period.

[0164] Here, for example, weather radar information is useful for detecting the presence or absence of clouds in the sky. However, while weather radar information can roughly detect the presence or absence of clouds and rainfall, it cannot accurately detect the presence or absence of rainfall within the target flight area. This is because weather radar may not be able to detect low-altitude clouds due to the cloud height or obstacles such as surrounding mountains.

[0165] In contrast to this, for example, by adding information obtained by actually observing the target flight area (for example, weather observation information included in the status observation information from the drone 11 or weather observation information from the weather observation device 13), the accuracy of estimating areas where there is no rain and only cloudy weather can be improved.

[0166] Furthermore, for example, the weather prediction unit 121 may predict the progression of cloudy areas by linearly interpolating the flow of clouds based on the flow of clouds obtained by actually observing the target flight area, or by predicting the flow of clouds using AI.

[0167] Here, a specific example of a method for estimating cloudy areas within a target flight area based on image data acquired by the imaging unit 52 of the drone 11 will be described.

[0168] For example, if the imaging unit 52 of the drone 11 is equipped with a stereo camera, it is possible to estimate the distance to an object in the image based on the acquired image data.

[0169] For example, if the imaging unit 52 of the drone 11 is equipped with a monocular camera rather than a stereo camera, the drone 11 itself moves, and parallax information can be obtained depending on the position of the drone 11. Therefore, it is possible to estimate the distance to an object in the image, just as in the case of a stereo camera.

[0170] As described above, the sensor unit 51 of the drone 11 is equipped with a GNSS receiver and is capable of detecting position information of the drone 11. Therefore, the drone 11 can associate the position of the drone 11 within the target flight area with the positions of objects on the ground in the image and the positions of clouds in the sky.

[0171] In addition, the range of the ground that can be photographed in one frame of a surveying camera (with one opening and closing of the shutter (during one exposure)) is determined by the flight altitude of the drone 11 and the angle of view of the camera.

[0172] For example, the photographing range C1 in FIG. 13A shows an example of the range on the ground that can be photographed in one frame of a surveying camera.

[0173] In contrast to this, one possible method for estimating the weather within the target flight area is to estimate it based on the range of the ground that can be photographed in one frame (hereinafter referred to as the ground photographing unit).

[0174] For example, the area on the ground corresponding to the target flight area may be divided into a plurality of unit areas for each ground capture unit, and the weather in each unit area may be estimated based on the brightness of pixels in the image corresponding to each unit area. For example, a unit area in which the percentage of pixels exceeding a predetermined brightness exceeds a predetermined threshold (e.g., 20%) may be estimated to be a clear-weather area. This threshold may be determined, for example, by AI (artificial intelligence).

[0175] It is also possible to predict the weather based on images of clouds in the sky, for example.

[0176] For example, it is possible to calculate the sky range corresponding to the ground shooting unit (hereinafter referred to as the projected shooting unit). For example, Fig. 14 shows an example of a projected shooting unit C3 corresponding to the ground shooting unit C2.

[0177] Alternatively, the weather in each unit area may be predicted based on the amount of clouds in the image of the projection shooting unit corresponding to each ground shooting unit.

[0178] For example, the Japan Meteorological Agency defines a clear sky as a state in which the proportion of clouds in the entire sky is less than 20%, a cloudy state as a state in which the proportion of clouds in the entire sky is more than 80%, and any other state as clear. For example, in accordance with this standard, the weather of a unit area corresponding to a projected shooting unit may be predicted based on the proportion of clouds in the image of the projected shooting unit.

[0179] The location of solar radiation on the ground changes depending on the position of clouds and the sun in the sky. In response to this, for example, the position of the projection shooting unit corresponding to the terrestrial shooting unit may be corrected based on the position of the terrestrial shooting unit and the position of the sun calculated based on the date and time.

[0180] Furthermore, when predicting the weather of each unit area, predictions other than sunny and cloudy may be made. For example, predictions of weather including rain, snow, and fog may be made based on the detection results of falling rain or snow, or the detection results of accumulating fog. For example, predictions of weather including lightning may be made based on the recognition results of thunderclouds based on the color of clouds, or the detection results of lightning.

[0181] For example, the sunlight conditions for a ground shooting unit, the amount of clouds for a projection shooting unit, and the rain or snow falling conditions may be detected based on images other than visible light images (e.g., infrared images) or sensor data.

[0182] Furthermore, the terrestrial photography units and projection photography units may be divided into tiles, for example, as shown in B of Figure 13, or adjacent terrestrial photography units and projection photography units may overlap, as shown in C of Figure 13.

[0183] Furthermore, when predicting the weather in a unit area, prediction results of surrounding unit areas may be used. In this case, the prediction results of the surrounding unit areas may be added with or without weighting.

[0184] In step S55, the planner 122 determines whether the drone 11 will pass through a non-cloudy area based on the time series movement of the drone 11 on the flight path in the current execution plan and the time series transition of non-cloudy areas within the target flight area estimated by the weather estimation unit 121. If it is determined that the drone 11 will pass through a non-cloudy area, the process proceeds to step S56.

[0185] In step S56, the planner 122 prepares a revised execution plan.

[0186] Specifically, the planner 122 creates a proposed correction to the execution plan based on the weather trends in unexecuted areas where photography has not yet been performed, the landing point and the weather trends leading to the landing point, the remaining battery charge of the drone 11, the amount of fluctuation in the time required and power consumption of the drone 11 due to route changes, etc. For example, the planner 122 corrects the flight path, the task execution period, and the timing of photography in order to avoid performing tasks (photography) in non-cloudy areas that are unsatisfactory areas as much as possible.

[0187] Here, an example of a method for correcting an execution plan will be described with reference to FIGS.

[0188] 15A shows the execution plan before correction and the weather conditions in the target flight area A1 before the start of task execution. This execution plan is the same as the execution plan described above with reference to FIG. 8, and the same flight route R1 as in FIG. 8 is set. Furthermore, area A11 within the target flight area A1 is cloudy.

[0189] FIG. 15B shows the weather conditions within the target flight area A1 when the drone 11 flies along flight path R1a, which is the first half of flight path R1, and reaches the upper left corner of area A12.

[0190] In addition, in B of Fig. 15, area A11 in A of Fig. 15 is shown divided into areas A12 to A15. Area A12 is approximately the right half of area A11. Area A15 is approximately the left half of area A11. Area A13 and area A14 are areas sandwiched between area A12 and area A15. Area A13 and area A14 are aligned vertically in a straight line, with area A13 located on the upper side and area A14 located on the lower side. Area A13 is vertically longer than area A14.

[0191] At this point, areas A12, A14, and A15 are cloudy, and area A13 is sunny.

[0192] 16 to 19 show correction proposals for the execution plan created in the state B of FIG. 15, respectively.

[0193] In the proposed correction in Figure 16, the flight path is not changed and flight continues as planned to the final destination, but photography is not performed in the non-cloudy areas. After reaching the final destination, photography is performed in the non-photography areas only after the weather becomes cloudy in the non-photography areas.

[0194] Specifically, as shown in A of Figure 16, the drone 11 moves to the final point Pe1 along flight routes R1b and R1c, which are the latter half of flight route R1. During this time, the drone 11 passes through area A13, which is sunny, without taking any images, and takes images as originally planned in areas A14 and A15, which are cloudy.

[0195] The drone 11 then waits at the final point Pe1 until the weather in the unphotographed area A13 turns cloudy. Then, as shown in B of Figure 16, after the weather in area A13 turns cloudy, the drone 11 follows the flight path R11 to area A13, photographs the area while moving from bottom to top within area A13, and then returns to the takeoff point Ps1. At this time, the flight path R11 is set so that the flight distance is as short as possible.

[0196] Note that a location other than the final point Pe1 may be set as the standby location based on predicted weather conditions, etc. For example, if it is predicted that it will rain at the final point Pe1 before the weather in area A13 turns cloudy, a location that is predicted not to rain may be set as the standby location. For example, if the weather in area A13 is sunny, the drone 11 may first move to area A13 and wait there.

[0197] In the proposed correction in Figure 17, the flight path is changed to bypass the non-cloudy area. After the aircraft reaches the end point of the changed flight path, the area where no image capture is being performed (hereinafter referred to as the non-image capture area) becomes cloudy, and then image capture is performed in the non-image capture area.

[0198] 17A, the drone 11 follows flight path R21, crosses area A13 from right to left, and then shifts slightly from right to left within area A15, alternately moving between a vertical path from top to bottom and a vertical path from bottom to top, until it finally arrives at final point Pe21 in the upper left corner of area A15.

[0199] The drone 11 then waits at the final point Pe21 until the weather in the unphotographed area A13 turns cloudy. Then, as shown in B of Figure 17, after the weather in area A13 turns cloudy, the drone 11 follows flight path R22 to area A13, photographs the area A13 and area A14 while moving downward, and then returns to the takeoff point Ps1. At this time, the flight path R22 is set so that the flight distance is as short as possible.

[0200] As with the example of the proposed amendment in FIG. 16, a location other than the final point Pe21 may be set as the waiting location based on the predicted weather conditions or the like.

[0201] In the proposed correction in Figure 18, the flight path is not changed, and photography is carried out as usual in cloudy areas, and the drone 11 waits around or inside non-cloudy areas until the weather becomes cloudy (until the weather conditions are met), and then movement and photography resumes after the weather changes to cloudy.

[0202] Specifically, as shown in A of Fig. 18, the drone 11 enters area A13, which is a non-cloudy area, along flight path R1b and waits at point Pp1 immediately after entering. After that, as shown in B of Fig. 18, after area A13 changes to cloudy weather, the drone 11 takes photographs while moving along flight path R1c, which is the latter half of flight path R1.

[0203] This correction plan is recommended, for example, when it is predicted that a non-cloudy area will change to cloudy weather early, or when the drone 11 has sufficient battery power remaining.

[0204] In the proposed correction in Figure 19, as with the proposed correction in Figure 16, the flight path is not changed and the flight continues as planned to the final point, but no photography is performed in the non-cloudy area. After reaching the final point, the drone 11 returns to the takeoff point and lands. After that, when the non-photography area becomes cloudy, photography is performed in the non-photography area.

[0205] Specifically, as shown in A of Fig. 19, similar to A of Fig. 16, the drone 11 continues to move to the final point Pe1 along flight routes R1b and R1c, which are the latter half of flight route R1. During this time, the drone 11 passes through area A13, which is sunny, without taking any images, and takes images as originally planned in areas A14 and A15, which are cloudy.

[0206] Next, as shown in B of Figure 19, after reaching the final point Pe1, the drone 11 returns to the takeoff point Ps1 along the flight path R1d and lands.

[0207] The drone 11 then waits at the takeoff point Ps1 until the weather in the unphotographed area A13 turns cloudy. Then, as shown in C of Figure 19, after the weather in area A13 turns cloudy, the drone 11 follows flight path R31 to fly to area A13, photographs the area while moving downward within area A13, and then returns to the takeoff point Ps1. At this time, the flight path R31 is set, for example, so that the flight distance is as short as possible.

[0208] This proposed correction is recommended, for example, when it is predicted that a non-cloudy area will change to cloudy slowly, or when there is little remaining battery power in the drone 11. This proposed correction also suggests, for example, a scheduled time to resume task execution (flight).

[0209] In step S57, the output unit 103, under the control of the output control unit 123, presents a proposed correction to the execution plan.

[0210] Here, an example of a method for presenting a proposed correction to an execution plan will be described with reference to FIGS. 20 and 21. FIG.

[0211] FIG. 20 shows an example of an execution status display screen that is displayed during normal task execution and indicates the execution status of a task.

[0212] In this example, a map 401 is displayed, which includes the target flight area and shows a map of the area in which the task is to be performed. The flight path is superimposed on the map 401, and an icon 402 indicating the current position of the drone 11 is also displayed.

[0213] 21 shows an example of an execution status display screen when a proposed correction to the execution plan is presented. This execution status display screen includes execution plan information showing the proposed correction to the execution plan.

[0214] Buttons 421a to 421d are arranged vertically on the left edge of this execution status display screen. Buttons 421a to 421d correspond to execution plan correction proposals 1 to 4, respectively. The contents of the execution plan correction proposal corresponding to the button selected with the cursor among buttons 421a to 421d are displayed in a window 424 on the right.

[0215] Hereinafter, when there is no need to distinguish between the buttons 421a to 421d, they will be simply referred to as buttons 421.

[0216] A scroll bar 422 is displayed to the right of the buttons 421a to 421d. The scroll bar 422 is used to scroll the displayed buttons 421 when there are a large number of correction proposals in the execution plan.

[0217] A window 423 is displayed below the buttons 421a to 421d. A screen similar to the execution status display screen in Fig. 20 is displayed in a reduced size within the window 423. In other words, the execution status of the current task is displayed within the window 423.

[0218] A map 431 is located in the center of the window 424. The map 431 includes the target flight area and shows a map of the area near the area where the task will be executed. Information indicating a proposed correction to the selected execution plan (Execution Plan 3) is displayed in the upper center of the map 431. The current date (June 1, 2023) is displayed in the upper right corner of the map 431.

[0219] Below the map 431 are arranged a timeline bar 432 and a playhead 433 for specifying the time at which information is to be displayed within the map 431. Below and in the center of the timeline bar 432, a play button 434 and a stop button 435 are arranged side by side.

[0220] The time (10:14) when the execution plan was revised is displayed above the left end of the timeline bar 432. The end time (10:30) of the task execution period is displayed above the right end of the timeline bar 432. The specified time (10:17) specified by the playhead 433 is displayed above the playhead 433.

[0221] The map 431 overlays the flight path for the selected revised execution plan and the predicted weather for the target flight area at the specified time. In this example, the cloudy, sunny, and rainy areas within the target flight area at the specified time are displayed in different display styles to show the weather distribution within the target flight area. Additionally, areas for which photography has already been completed (task completion areas) are displayed in a different display style from the other areas, and gradually expand as the task progresses. A legend indicating the display style of each area is displayed in the lower right corner of the map 431.

[0222] An icon 436 indicating the position of the drone 11 at the specified time is superimposed on the map 431. At the top left edge of the map 431, the weather (cloudy) at the position of the drone 11 indicated by the icon 436 is displayed.

[0223] For example, when the play button 434 is pressed, the playhead 433 shifts from left to right on the timeline bar 432, and the specified time advances. In addition, as the specified time advances, the display of the weather distribution on the map 431 changes, and an icon 436 indicating the position of the drone 11 moves along the flight path.

[0224] On the other hand, when the stop button 435 is pressed, the play button 434 stops, and an icon 436 indicating the weather distribution at the specified time when the play button 434 was stopped and the position of the drone 11 is displayed on the map 431. It is also possible to directly slide the play button 434 on the timeline bar 432 to specify a specified time.

[0225] This allows the user to easily check the flight path in the revised proposal for each execution plan, as well as the time-series changes in the weather distribution and the time-series movements of the drone 11 in the revised proposal for each execution plan.

[0226] In step S58, the controller 12 initiates the execution of the selected correction proposal.

[0227] For example, the user selects a desired correction proposal from the presented correction proposals via the input unit 101 .

[0228] In response to this, the input unit 101 supplies the input information to the information processing unit 102. The planning unit 122 generates execution plan information for causing the drone 11 to execute the selected correction plan. The operation control unit 124 transmits the execution plan information to the drone 11 via the communication unit 104.

[0229] In response to this, the information processing unit 53 of the drone 11 receives the execution plan information via the communication unit 55. The operation control unit 74 starts controlling the imaging unit 52 and the power unit 54 based on the execution plan information. As a result, the drone 11 starts executing the task in accordance with the corrected execution plan.

[0230] For example, if the user selects the current execution plan or if no correction plan is selected, the drone 11 continues with the current execution plan as is.

[0231] Thereafter, the process proceeds to step S59.

[0232] On the other hand, if it is determined in step S55 that the drone 11 will not pass through the non-cloudy area, the processing of steps S56 to S58 is skipped and the processing proceeds to step S59.

[0233] If it is determined in step S53 that the execution plan is not to be corrected in real time, steps S54 to S58 are skipped and the process proceeds to step S59.

[0234] In step S59, the operation control unit 124 determines whether the task has been completed. If it is determined that the task has not been completed, the process returns to step S53.

[0235] Thereafter, the processes of steps S53 to S59 are repeatedly executed until it is determined in step S59 that the task is completed.

[0236] On the other hand, if it is determined in step S59 that the task is completed, the task execution control process ends.

[0237] In this way, an execution plan is created and corrected so that surveying photographs are taken under cloudy skies whenever possible, reducing the probability of surveying photographs failing and avoiding the need to redo the surveying photographs. This avoids the increase in time required for preparing and executing the surveying photographs again, as well as the transfer of data. It also avoids an increase in the storage capacity of the photograph data.

[0238] Furthermore, since the three-dimensional data is generated based on high-quality photographs taken under cloudy skies, the accuracy of the three-dimensional data is improved.

[0239] Furthermore, the user can easily check the contents of the proposed corrections to the execution plan and the weather trends, and can therefore easily select a proposed correction that satisfies the user.

[0240] <<2. Modifications>> Modifications of the above-described embodiments of the present technology will now be described.

[0241] <Modifications of Processing Allocation> The above-described processing allocation can be changed as appropriate.

[0242] As described above, the process of estimating the weather in the target flight area can be performed by any of the drone 11, the controller 12, and the server 14. Also, for example, two or more of the drone 11, the controller 12, and the server 14 may cooperate or share the work to perform the process of estimating the weather in the target flight area.

[0243] As described above, the creation or correction process of the execution plan can be executed by any of the drone 11, the controller 12, and the server 14. Furthermore, for example, two or more of the drone 11, the controller 12, and the server 14 may cooperate or share the work to execute the creation or correction process of the execution plan.

[0244] For example, two or more drones 11 may cooperate or share the work to estimate the weather in the target flight area, or to create or correct an execution plan.

[0245] <Modifications on the Creation and Correction Method of the Execution Plan> The creation method of the proposed correction of the execution plan and the number of corrections to be created can be changed as appropriate. For example, the proposed correction may be created by changing the conditions used to create the proposed correction of the execution plan.

[0246] For example, weather conditions suitable for executing a task and weather conditions unsuitable for executing a task differ depending on the type of task to be executed.

[0247] For example, when taking aerial photographs for purposes other than surveying, it is expected that sunny days may be more suitable than cloudy days. For example, when observing vegetation indices using the drone 11, sunny or cloudy days are suitable. For example, when spraying pesticides using the drone 11, sunny days are suitable. Note that although pesticides can be sprayed without any particular problems even on cloudy days, sunny days are preferable because there is a risk that the pesticides will be washed away if it rains after spraying. For example, when transporting cargo using the drone 11, sunny or cloudy days are suitable.

[0248] In response to this, for example, regardless of the type of task, the execution plan may be created or corrected so as to increase the probability that weather conditions suitable for executing the task are met at the location where the drone 11 executes the task. Specifically, for example, at least one of the flight path and the execution period may be set so as to increase the probability that weather conditions suitable for executing the task are met at the location where the drone 11 executes the task.

[0249] Weather conditions suitable for performing a task may be set in advance, may be set based on the task information described above, or may be set by the user, for example.

[0250] Other Modifications The present technology can be applied to autonomous flying objects other than drones that fly autonomously according to an execution plan and can perform various tasks. For example, the present technology can be applied to flying cars, etc.

[0251] <<3. Others>> <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.

[0252] FIG. 22 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0253] In the computer 1000 , a CPU (Central Processing Unit) 1001 , a ROM (Read Only Memory) 1002 , and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004 .

[0254] An input / output interface 1005 is further connected to the bus 1004. An input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010 are connected to the input / output interface 1005.

[0255] The input unit 1006 includes input switches, buttons, a microphone, an image sensor, etc. The output unit 1007 includes a display, a speaker, etc. The storage unit 1008 includes a hard disk, a non-volatile memory, etc. The communication unit 1009 includes a network interface, etc. The drive 1010 drives removable media 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0256] In the computer 1000 configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program recorded in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0257] The program executed by the computer 1000 (CPU 1001) can be provided by being recorded on a removable medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0258] In the computer 1000, the program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting the removable medium 1011 into the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be installed in the ROM 1002 or the storage unit 1008 in advance.

[0259] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0260] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0261] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0262] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0263] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0264] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0265] <Examples of Combinations of Configurations> The present technology can also have the following configurations.

[0266] (1) An information processing device comprising: a planning unit that creates an execution plan for the autonomous air vehicle to execute a task, the execution plan including a flight path of the autonomous air vehicle, based on predicted weather conditions in a target flight area in which the autonomous air vehicle will fly and weather conditions suitable for the task to be executed by the autonomous air vehicle. (2) The information processing device described in (1), wherein the planning unit corrects the execution plan based on predicted weather conditions in the target flight area when the autonomous air vehicle is executing the task in accordance with the execution plan. (3) The information processing device described in (2), wherein the planning unit corrects the execution plan to avoid executing the task in an unsatisfied area, which is an area that does not satisfy the weather conditions. (4) The information processing device described in (3), wherein the planning unit corrects the execution plan to execute the task in the unsatisfied area when the unsatisfied area satisfies the weather conditions. (5) The information processing device described in (4), wherein the planning unit corrects the execution plan so that the autonomous air vehicle waits around or within the unsatisfied area until the unsatisfied area satisfies the weather conditions. (6) The information processing device according to any of (3) to (5), wherein the planning unit corrects the flight path so as to detour around the unsatisfied area. (7) The information processing device according to any of (2) to (6), further comprising: an output control unit that controls the presentation of execution plan information indicating a proposed correction to the execution plan, the execution plan information including the weather distribution within the target flight area and the flight path. (8) The information processing device according to (7), wherein the execution plan information further includes the position of the autonomous air vehicle and the area where the task has been completed. (9) The execution plan includes an execution period for executing the task, and the planning unit sets at least one of the flight path and the execution period so as to increase the rate at which the weather conditions are satisfied at the position where the autonomous air vehicle executes the task. (10) The information processing device according to any of (9), further comprising: an output control unit that controls the presentation of multiple execution plans in which at least one of the flight path and the execution period differs.(11) The information processing device according to (1), further comprising an output control unit that controls presentation of execution plan information indicating the execution plan, including the weather distribution within the target flight area and the flight path. (12) The execution plan information includes information indicating the time series progression of the weather within the target flight area and the time series movement of the autonomous air vehicle. (13) The information processing device according to any of (1) to (12), further comprising a weather prediction unit that predicts the weather in the target flight area based on weather forecast information in the vicinity of the target flight area. (14) The information processing device according to (13), further predicts the weather in the target flight area based on meteorological observation information indicating observation results of the weather in the vicinity of the target flight area. (15) The meteorological observation information includes information observed by the autonomous air vehicle while it is executing the task. (16) The information processing device according to any of (1) to (15), further comprising an operation control unit that causes the autonomous air vehicle to execute the task in accordance with the execution plan. (17) The information processing device according to any one of (1) to (16), wherein the task is taking photographs for surveying, and the weather conditions suitable for the task are cloudy. (18) An information processing method, wherein an information processing device creates an execution plan for the autonomous air vehicle to execute the task, the execution plan including a flight path of the autonomous air vehicle, based on a predicted weather result for a flight area in which the autonomous air vehicle will fly and weather conditions suitable for the task to be executed by the autonomous air vehicle. (19) An information processing system, comprising: a weather prediction unit that predicts the weather in the flight area based on weather forecast information for the vicinity of the flight area in which the autonomous air vehicle will fly; a planning unit that creates an execution plan for the autonomous air vehicle to execute the task, the execution plan including the flight path of the autonomous air vehicle, based on the predicted weather result for the flight area and weather conditions suitable for the task to be executed by the autonomous air vehicle; and an operation control unit that causes the autonomous air vehicle to execute the task in accordance with the execution plan, wherein the weather prediction unit, the planning unit, and the operation control unit are each disposed in any one of two or more information processing devices.

[0267] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0268] 1 Information processing system, 11 Drone, 12 Controller, 13A, 13B Weather observation device, 14 Server, 51 Sensor unit, 52 Photography unit, 52F to 52O Camera, 53 Information processing unit, 71 Observation unit, 72 Weather prediction unit, 73 Planning unit, 74 Operation control unit, 102 Information processing unit, 103 Output unit, 121 Weather prediction unit, 122 Planning unit, 123 Output control unit, 124 Operation control unit, 158 Information processing unit, 171 Observation unit, 201 Information processing unit, 221 Information collection unit, 222 Weather prediction unit, 223 Planning unit

Claims

1. An information processing device comprising: a planning unit that creates an execution plan for an autonomous air vehicle to execute a task, the execution plan including a flight path of the autonomous air vehicle, based on predicted weather conditions in a flight target area in which the autonomous air vehicle will fly and weather conditions suitable for the task to be executed by the autonomous air vehicle.

2. The information processing device according to claim 1, wherein the planning unit corrects the execution plan based on a predicted result of weather in the target flight area when the autonomous air vehicle is executing the task in accordance with the execution plan.

3. The information processing device according to claim 2, wherein the planning unit corrects the execution plan so as to avoid execution of the task in an unsatisfied area, which is an area that does not satisfy the weather conditions.

4. The information processing device according to claim 3, wherein the planning unit corrects the execution plan so that the task is executed in the unsatisfied area when the unsatisfied area satisfies the weather condition.

5. The information processing device according to claim 4, wherein the planning unit corrects the execution plan so that the autonomous air vehicle waits around or within the unsatisfied area until the unsatisfied area satisfies the weather conditions.

6. The information processing device according to claim 3, wherein the planning unit corrects the flight path so as to bypass the unsatisfied area.

7. An information processing device as described in claim 2, further comprising an output control unit that controls the presentation of execution plan information including the weather distribution within the target flight area and the flight path and indicating a correction proposal for the execution plan.

8. The information processing device according to claim 7, wherein the execution plan information further includes the position of the autonomous flying vehicle and the area in which the task has been completed.

9. The information processing device of claim 1, wherein the execution plan includes an execution period for executing the task, and the planning unit sets at least one of the flight path and the execution period so as to increase the rate at which the weather conditions are satisfied at the location where the autonomous air vehicle executes the task.

10. The information processing device according to claim 9, further comprising an output control unit that controls the presentation of multiple execution plans in which at least one of the flight path and the execution period is different.

11. The information processing device according to claim 1, further comprising an output control unit that controls the presentation of execution plan information indicating the execution plan, including the weather distribution within the target flight area and the flight route.

12. The information processing device according to claim 11, wherein the execution plan information includes information indicating the time series changes in weather within the target flight area and the time series movement of the autonomous flying vehicle.

13. The information processing device according to claim 1, further comprising a weather prediction unit that predicts the weather in the target flight area based on weather forecast information in the vicinity of the target flight area.

14. The information processing device according to claim 13, further comprising: an information processing device for predicting the weather in the target flight area based on meteorological observation information indicating the observation results of the weather in the vicinity of the target flight area.

15. The information processing device according to claim 14, wherein the meteorological observation information includes information observed by the autonomous flying object while executing the task.

16. The information processing device according to claim 1, further comprising an operation control unit that causes the autonomous air vehicle to execute the task in accordance with the execution plan.

17. The information processing device according to claim 1, wherein the task is taking photographs for surveying, and the weather conditions suitable for the task are cloudy skies.

18. An information processing method in which an information processing device creates an execution plan for the autonomous air vehicle to execute a task, including a flight path for the autonomous air vehicle, based on predicted weather conditions in a flight area in which the autonomous air vehicle will fly and weather conditions suitable for the task to be executed by the autonomous air vehicle.

19. An information processing system comprising: a weather prediction unit that predicts the weather in a target flight area in which an autonomous air vehicle flies based on weather forecast information for the vicinity of the target flight area; a planning unit that creates an execution plan for the autonomous air vehicle to execute a task, including a flight path for the autonomous air vehicle, based on the predicted weather in the target flight area and weather conditions suitable for a task to be executed by the autonomous air vehicle; and an operation control unit that causes the autonomous air vehicle to execute the task in accordance with the execution plan, wherein the weather prediction unit, planning unit, and operation control unit are each disposed in one of two or more information processing devices.

Images