Information processing device, information processing method, and information processing system

Abstract

The present technology relates to an information processing device, an information processing method, and an information processing system that facilitate the determination of the validity of an execution plan for an autonomous mobile body to execute a mission. The information processing device according to the present technology is provided with an output control unit that controls the display of: execution plan information related to an execution plan for an autonomous mobile body to execute a mission; area information related to an execution area in which the mission is executed; and additional information that can be applied to determination of the validity of the execution plan. The present technology can be applied to, for example, an information processing device that sets a mission to be executed by a drone.

Description

Information processing device, information processing method, and information processing system

[0001] The present technology relates to an information processing device, an information processing method, and an information processing system, and particularly to an information processing device, an information processing method, and an information processing system that make it easier to determine the validity of a travel path of an autonomous moving body.

[0002] A technology has been proposed that allows users to confirm the validity of a drone's flight path by generating a flight path image that includes the flight path depicted in a virtual space and displaying a composite image on a user's terminal that is superimposed on a captured image obtained based on shooting status information (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2021-104802

[0004] Meanwhile, in recent years, drones have been flying autonomously and autonomously executing missions to photograph designated areas, buildings, etc.

[0005] As a result, there is a need to make it easier to determine the validity of execution plans for drones to carry out missions.

[0006] This technology was developed in light of these circumstances, and makes it easier to determine the validity of an execution plan for an autonomous mobile object such as a drone to carry out a mission.

[0007] An information processing device according to a first aspect of the present technology includes an output control unit that controls the display of execution plan information regarding an execution plan for an autonomous moving body to execute a mission, area information regarding an execution area in which the mission will be executed, and additional information that can be applied to determining the validity of the execution plan.

[0008] In the information processing method according to the first aspect of the present technology, an information processing device controls the display of execution plan information relating to an execution plan for an autonomous moving body to execute a mission, area information relating to an execution area in which the mission will be executed, and additional information applicable to determining the validity of the execution plan.

[0009] An information processing system according to a second aspect of the present technology includes an execution plan creation unit that creates an execution plan for an autonomous moving body to execute a mission, based on mission information related to a mission to be executed by the autonomous moving body and area information related to an execution area in which the mission will be executed; an additional information generation unit that generates additional information applicable to determining the validity of the execution plan, based on the execution plan information related to the execution plan and the area information; and an output control unit that controls display of the execution plan information, the area information, and the additional information.

[0010] In a first aspect of the present technology, display of execution plan information regarding an execution plan for an autonomous moving body to execute a mission, area information regarding an execution area in which the mission is to be executed, and additional information applicable to determining the validity of the execution plan is controlled.

[0011] In a second aspect of the present technology, an execution plan for an autonomous moving body to execute a mission is created based on mission information related to a mission to be executed by the autonomous moving body and area information related to an execution area in which the mission is to be executed, additional information applicable to determining the validity of the execution plan is generated based on execution plan information related to the execution plan and the area information, and display of the execution plan information, the area information, and the additional information is controlled.

[0012] 1 is a block diagram showing an embodiment of an information processing system to which the present technology is applied; FIG. 2 is a block diagram showing an example of the configuration of the functions of a user terminal; FIG. 3 is a block diagram showing an example of the configuration of the functions of a server; FIG. 4 is a flowchart for explaining execution plan creation processing; FIG. 5 is a diagram showing an example of a mission setting screen; FIG. 6 is a diagram showing an example of a mission setting screen; FIG. 7 is a diagram showing an example of a display of an execution plan; FIG. 8 is a diagram showing an example of a display of an execution plan; FIG. 9 is a diagram showing an example of a display of an execution plan; FIG. 10 is a diagram showing an example of a display of an execution plan; FIG. 11 is a diagram showing an example of a display of an execution plan; FIG. 12 is a diagram showing an example of a display of an execution plan; FIG. 13 is a diagram showing an example of a display of an execution plan;

[0013] Hereinafter, embodiments of the present technology will be described. The description will be made in the following order: 1. Embodiment 2. Modification 3. Other

[0014] <<1. Embodiment>> An embodiment of the present technology will be described with reference to FIGS. 1 to 15 .

[0015] <Configuration example of information processing system 1> First, a configuration example of an information processing system 1 to which the present technology is applied will be described with reference to Fig. 1. The information processing system 1 is a system that, for example, creates an execution plan for a drone 11 to execute a given mission (command) and controls the flight operation of the drone 11 in accordance with the execution plan.

[0016] Here, the content of the mission is not particularly limited as long as the drone 11 can execute it autonomously. For example, the drone 11 is given a mission to photograph a designated area or object (e.g., a building, structure, etc.) for the purpose of inspection, surveying, etc. Specifically, the drone 11 is given a mission to comprehensively photograph a designated area (hereinafter referred to as the target photographing area). For example, the drone 11 is given a mission to comprehensively photograph the surface of a designated object (hereinafter referred to as the target photographing object). For example, the drone 11 is given a mission to move to each photographing location according to a list of designated photographing locations (hereinafter referred to as the photographing location list) and photograph the image.

[0017] Hereinafter, the photographing target area and the photographing target object will be collectively referred to as the photographing target. Note that in the case of a mission in which photographing is performed according to a photographing location list, by specifying the photographing location, the photographing target area or photographing target object photographed from each photographing location is indirectly specified as the photographing target.

[0018] The information processing system 1 includes a drone 11, a controller 12, a user terminal 13, and a server 14.

[0019] The drone 11, the user terminal 13, and the server 14 are connected to each other and can communicate via a network 21. The network 21 includes various networks such as the Internet and a mobile phone network, base stations, access points, and the like.

[0020] The drone 11 can directly communicate wirelessly with the controller 12 using a predetermined communication method. For example, a 2.4 GHz spread spectrum method is used as the communication method between the drone 11 and the controller 12.

[0021] The drone 11 can connect to the network 21, for example, by wireless communication such as Wi-Fi or mobile communication such as 4G or 5G. Therefore, the drone 11 can communicate with the user terminal 13, the server 14, etc. via the network 21.

[0022] The drone 11 is equipped with photographic equipment such as a camera, and is capable of photographing the surroundings of the drone 11.

[0023] The drone 11 can fly by remote control using the controller 12 or the user terminal 13. The drone 11 can also fly autonomously without being remotely controlled by the controller 12 or the user terminal 13. For example, the drone 11 can execute a given mission while flying autonomously in accordance with execution plan information provided by the controller 12, the user terminal 13, or the server 14.

[0024] The controller 12 is configured, for example, by a proportional system. As described above, the controller 12 is capable of directly communicating wirelessly with the drone 11 and is used to remotely control the drone 11.

[0025] The controller 12 can use a predetermined communication method to directly communicate wirelessly or via wire with the user terminal 13. The communication method between the controller 12 and the user terminal 13 may be, for example, a short-range wireless communication method such as Bluetooth (registered trademark) or a wired communication method using a USB (Universal Serial Bus) cable or the like.

[0026] The user terminal 13 is configured by an information processing terminal such as a smartphone, a tablet terminal, a PC (personal computer), etc. The user terminal 13 can connect to the network 21 by wireless communication such as Wi-Fi, or by mobile communication such as 4G or 5G. Therefore, the user terminal 13 can communicate with the drone 11, the server 14, etc. via the network 21.

[0027] By executing a specific application, the user terminal 13 can create an execution plan for the drone 11 to carry out a mission given by the user, and can remotely control the drone 11, etc.

[0028] The server 14 is capable of communicating with the drone 11 and the user terminal 13 via the network 21. The server 14 provides, for example, information necessary for the user terminal 13 to create an execution plan.

[0029] Although not shown in the drawings, the information processing system 1 is provided with a plurality of sets of drones 11, controllers 12, and user terminals 13. The number of drones 11, controllers 12, and user terminals 13 in each set can be set arbitrarily. For example, a single drone 11 may be provided with a plurality of controllers 12, or a single user terminal 13. For example, a single controller 12 may be provided with a plurality of drones 11, or a single user terminal 13 may be provided with a plurality of drones 11. For example, either the controller 12 or the user terminal 13 may be omitted.

[0030] FIG. 2 shows an example of the functional configuration of the user terminal 13.

[0031] The user terminal 13 includes an input unit 101 , a control unit 102 , an output unit 103 , a communication unit 104 , and a storage unit 105 .

[0032] The input unit 101 includes various input devices and operation devices, and is used to input various information and operate the user terminal 13. The input unit 101 supplies the control unit 102 with input information input using the input devices or operation devices.

[0033] The control unit 102 is configured with a processor such as a CPU, a memory, etc. The control unit 102 executes a predetermined program to control each unit of the user terminal 13 and execute various processes. The control unit 102 includes an information acquisition unit 111, an execution plan creation unit 112, an output control unit 113, and a remote control unit 114.

[0034] The information acquisition unit 111 acquires various types of information from the outside via the communication unit 104. For example, the information acquisition unit 111 receives area information and additional information from the server 14 via the network 21 and the communication unit 104.

[0035] The area information includes information regarding the area in which the drone 11 will perform the mission (hereinafter referred to as the execution area).

[0036] For example, in the case of a mission to photograph a photographing target (a photographing target area or a photographing target object), the area including the photographing target becomes the execution area. For example, in the case of a mission to photograph according to a photographing location list, the execution area becomes each photographing location and the area including the photographing target to be photographed from each photographing location.

[0037] The additional information includes information that can be applied by the user to determine the validity of the execution plan created by the execution plan creation unit 112. In other words, the additional information includes auxiliary information that assists the user in determining the validity of the execution plan.

[0038] The execution plan creation unit 112 creates an execution plan for the drone 11 to execute the mission based on mission information regarding the mission provided by the user via the input unit 101, area information, and, if necessary, equipment information.

[0039] The equipment information includes, for example, information about the photographic equipment such as a camera equipped on the drone 11, and information about the drone 11 itself.

[0040] The execution plan creation unit 112 transmits execution plan information related to the execution plan and device information to the server 14 via the communication unit 104 and the network 21. The execution plan creation unit 112 transmits the execution plan information to the controller 12 via the communication unit 104, and transmits the execution plan information to the drone 11 via the communication unit 104 and the network 21.

[0041] The output control unit 113 controls the output of various types of information (for example, visual information, auditory information, and tactile information) from the output unit 103 .

[0042] The remote control unit 114 executes remote control of the flight operation of the drone 11. For example, the remote control unit 114 generates operation commands or control parameters for remotely controlling the drone 11, and transmits them to the drone 11 via the communication unit 104 and the network 21.

[0043] The output unit 103 includes various output devices that output visual information, auditory information, or tactile information. For example, the output unit 103 includes a display device, an audio output device, a haptic device, etc. The output unit 103 outputs various types of information under the control of the output control unit 113.

[0044] The communication unit 104 includes various communication devices and performs wireless communication with other devices using a predetermined communication method. For example, the communication unit 104 communicates with the drone 11, the server 14, etc. via the network 21. For example, the communication unit 104 performs direct wireless communication or wired communication with the controller 12.

[0045] The storage unit 105 includes, for example, a nonvolatile memory, and stores various types of information necessary for processing by the user terminal 13. For example, the storage unit 105 stores device information about the imaging equipment included in the drone 11 and the drone 11 itself.

[0046] <Configuration Example of Server 14> FIG. 3 shows a functional configuration example of the server 14. As shown in FIG.

[0047] The server 14 includes a control unit 151 , a communication unit 152 , and a storage unit 153 .

[0048] The control unit 151 is configured with a processor such as a CPU, a memory, etc. The control unit 151 executes a predetermined program to control each unit of the server 14 and execute various processes. The control unit 151 includes an information acquisition unit 161, an area information generation unit 162, and an additional information generation unit 163.

[0049] The information acquisition unit 161 acquires various types of information from the outside via the communication unit 152. For example, the information acquisition unit 161 receives execution plan information and device information from the user terminal 13 via the network 21 and the communication unit 152.

[0050] The area information generation unit 162 generates area information regarding an execution area in which the drone 11 executes a mission, based on at least one of map information, images, two-dimensional models, and three-dimensional models of various locations stored in the storage unit 153. The area information generation unit 162 transmits the area information to the user terminal 13 via the communication unit 152 and the network 21.

[0051] The additional information generating unit 163 generates additional information including information applicable to determining the validity of the execution plan based on the execution plan information, device information, and area information. The additional information generating unit 163 transmits the additional information to the user terminal 13 via the communication unit 152 and the network 21.

[0052] The communication unit 152 includes various communication devices and communicates with other devices using a predetermined communication method. For example, the communication unit 152 communicates with the drone 11, the user terminal 13, and the like via the network 21.

[0053] The storage unit 153 includes, for example, a non-volatile memory and stores various information necessary for processing by the server 14. For example, the storage unit 153 stores at least one of map information, images, two-dimensional models, and three-dimensional models of various locations. The map information includes, for example, two-dimensional or three-dimensional maps of various locations. The three-dimensional models are, for example, models that show the three-dimensional shapes of structures, terrain, etc. The two-dimensional models are, for example, models that show the two-dimensional shapes of structures, terrain, etc. near the execution area.

[0054] <Execution Plan Creation Process> Next, the execution plan creation process executed by the information processing system 1 will be described with reference to the flowchart of FIG.

[0055] This process is executed between the user terminal 13 and the server 14, for example.

[0056] In step S1, the user terminal 13 requests area information.

[0057] For example, the user inputs information about an execution area in which the drone 11 executes a mission via the input unit 101. The input unit 101 supplies input information including the input information to the control unit 102.

[0058] The information acquisition unit 111 generates area information request information for requesting area information related to the execution area, and transmits the area information request information to the server 14 via the communication unit 104 and the network 21.

[0059] In response to this, in step S2, the server 14 transmits the area information.

[0060] Specifically, the information acquisition unit 161 of the server 14 receives area information request information from the user terminal 13 via the network 21 and the communication unit 152 .

[0061] The area information generation unit 162 generates area information regarding the execution area based on the area information request information and at least one of map information, images, two-dimensional models, and three-dimensional models of various locations stored in the memory unit 153.

[0062] The area information includes information necessary for the user to set a mission and confirm an execution plan for executing the mission, for example, the area information includes at least one of map information, images, three-dimensional models, and two-dimensional models of the vicinity of the execution area.

[0063] The area information generating unit 162 transmits the area information to the user terminal 13 via the communication unit 152 and the network 21 .

[0064] In response to this, in step S3, the information acquisition unit 111 of the user terminal 13 receives the area information from the server 14 via the network 21 and the communication unit 104.

[0065] Although not shown in the flowchart, the processes of steps S1 to S3 are executed as appropriate, for example, every time the user terminal 13 needs new area information.

[0066] In step S4, the user terminal 13 accepts the setting of the mission.

[0067] For example, drone 11 is given a mission to photograph a desired area (area to be photographed) or the vicinity of the object (object to be photographed) in order to inspect or survey the area or object, or a mission to photograph the area to be photographed or the object to be photographed according to a list of photographing locations.

[0068] For example, under the control of the output control unit 113, the output unit 103 displays a setting screen (hereinafter referred to as the mission setting screen) for setting the mission to be executed by the drone 11.

[0069] FIG. 5 shows an example of a mission setting screen.

[0070] The mission setting screen is divided into an area information display section 201 and an operation section 202 .

[0071] The area information display section 201 displays area information around the execution area, for example, map information, images, three-dimensional models, or two-dimensional models around the execution area.

[0072] The operation unit 202 displays various operation parts for performing mission setting operations and the like.

[0073] For example, on the mission setting screen, the user can set the time-series movement of the drone 11. Specifically, for example, the user can set the time-series position (e.g., latitude, longitude, and altitude) and speed (e.g., horizontal speed and vertical speed) of the drone 11.

[0074] For example, on the mission setting screen, the user can set time-series shooting conditions for the photographing equipment such as a camera equipped on the drone 11 (hereinafter simply referred to as the photographing equipment of the drone 11). Specifically, for example, the user can set the area to be photographed, the object to be photographed, the photographing location where the photographing equipment of the drone 11 will take photographs, as well as the photographing direction, focus position, exposure, etc. of the photographing equipment of the drone 11.

[0075] In the following, when the photographing equipment of the drone 11 takes photographs, etc., it will be simply referred to as the drone 11 taking photographs, etc.

[0076] In step S5, the execution plan creation unit 112 of the user terminal 13 creates an execution plan based on the mission information related to the mission set by the user, the area information, and, if necessary, the device information. For example, the execution plan includes the start time, end time, flight path (movement path), flight speed, shooting conditions, etc.

[0077] It should be noted that the user does not necessarily have to set all items when setting a mission. Items not set by the user are automatically set when the execution plan creation unit 112 creates an execution plan, for example.

[0078] For example, the flight path may be set by the user, or may be automatically set by the execution plan creation unit 112 based on the mission set by the user.

[0079] For example, the mission setting screen in FIG. 5 shows an example in which a flight path 211 is set by the user.

[0080] For example, FIG. 6 shows an example in which, when a photographing area 251 is set by the user, the execution plan creation unit 112 automatically sets a flight path 211 so that the photographing area 251 can be photographed thoroughly.

[0081] 6 indicate the order of the flight path 252, and some parts are omitted from the illustration. A circled + is used to add a vertex to the photography target area 251. When the circular mark in the center of the photography target area 251 is dragged and dropped, the entire photography target area 251 moves.

[0082] Hereinafter, the flight route planned in the execution plan will be referred to as the planned route.

[0083] In step S6, the execution plan creation unit 112 of the user terminal 13 transmits the execution plan information and the device information via the communication unit 104 and the network 21.

[0084] The execution plan information includes information about the execution plan created in step S5.

[0085] The device information includes, for example, information about the photographic equipment such as a camera equipped on the drone 11 and the drone 11 itself. The information about the photographic equipment includes, for example, information about the model name, equipment, functions, performance, etc. of the photographic equipment. The information about the drone 11 itself includes, for example, information about the model name, equipment, functions, performance, etc. of the drone 11.

[0086] In response to this, in step S7, the information acquisition unit 161 of the server 14 receives the execution plan information and the device information via the network 21 and the communication unit 152.

[0087] In step S8, the additional information generation unit 163 of the server 14 generates additional information. Specifically, the additional information generation unit 163 generates additional information applicable to determining the validity of the execution plan based on the area information, the execution plan information, and the device information.

[0088] For example, the additional information generation unit 163 calculates the distance between the planned route of the drone 11 and objects around the planned route based on area information and execution plan information (particularly the planned route).

[0089] The additional information generation unit 163 sets sections on the planned route where there is a risk of the drone 11 colliding with or coming into contact with surrounding objects on the planned route and where caution is required (hereinafter referred to as "cautionary sections") based on the distance between the planned route and surrounding objects. For example, the additional information generation unit 163 sets sections on the planned route where the distance to surrounding objects is less than a threshold (hereinafter referred to as "approach determination threshold") as "cautionary sections."

[0090] In addition, the approach judgment threshold may be changed based on at least one of operational information such as the speed and acceleration of the drone 11, environmental information around the planned route (drone 11), and information about the drone 11's photographing equipment.

[0091] For example, the additional information generation unit 163 increases the approach judgment threshold as the speed of the drone 11 increases, and decreases the approach judgment threshold as the speed of the drone 11 decreases. For example, in a section where the drone 11 flies at a speed of 10 m / s, the approach judgment threshold is set to 10 m, and in a section where the drone 11 flies at a speed of 1 m / s, the approach judgment threshold is set to 1 m. As a result, for example, in a section where the drone 11 flies at a speed of 10 m / s, sections where the distance to surrounding objects is 10 m or less are set as caution zones. On the other hand, in a section where the drone 11 flies at a speed of 1 m / s, sections where the distance to surrounding objects is 1 m or less are set as caution zones.

[0092] For example, the additional information generation unit 163 changes the approach determination threshold based on predicted values ​​of the amount of sunlight, weather, wind speed, etc. around the drone 11. For example, when the drone 11 flies during the day, the approach determination threshold may be set to 10 m, and when the drone 11 flies at night, the approach determination threshold may be set to 5 m.

[0093] For example, the additional information generation unit 163 changes the approach determination threshold based on the focal length of the imaging device of the drone 11. For example, when the focal length is 50 mm, the approach determination threshold is set to 5 m because it becomes difficult to see the surroundings when getting too close to the subject. On the other hand, when the focal length is 24 mm, it is possible to get closer to the subject and take a photograph than when the focal length is 50 mm, so the approach determination threshold is set to 2 m.

[0094] For example, based on the area information, the execution plan information, and the equipment information, the additional information generation unit 163 calculates the shooting conditions at each shooting point of the drone 11. For example, the additional information generation unit 163 calculates the shooting range of the shooting equipment of the drone 11 at each shooting point, the shooting direction of the subject, the focus state of the subject, the resolution of the subject, etc.

[0095] The additional information generation unit 163 generates additional information including information regarding the distance between the planned route and surrounding objects, information regarding areas requiring caution, and information regarding the shooting conditions at each shooting point of the drone 11 (hereinafter referred to as shooting condition information).

[0096] In step S9, the additional information generating unit 163 of the server 14 transmits the additional information to the user terminal 13 via the communication unit 152 and the network 21.

[0097] In response to this, in step S10, the control unit 102 of the user terminal 13 receives the additional information from the server 14 via the network 21 and the communication unit 104.

[0098] In step S11, the output unit 103 of the user terminal 13, under the control of the output control unit 113, displays the execution plan together with additional information.

[0099] Here, examples of displaying an execution plan will be described with reference to FIGS.

[0100] 7 to 13 show examples of display of an execution plan when a mission to photograph a building 301 is given.

[0101] 7, a 3D model of the vicinity of a building 301 is displayed as area information, and a planned route 311 is superimposed on the area information. In addition, a message 312, a shooting range 313, a shooting range 314, and a menu 315 are superimposed on the area information as additional information.

[0102] The planned route 311 shows, by lines, the route along which the drone 11 is scheduled to fly in the execution plan. Within the planned route 311, sections requiring caution are displayed in a different manner from other sections. In this example, sections of the planned route 311 where the distance to the building 301 is less than the approach judgment threshold are displayed as caution sections using dotted lines. In this way, the display manner of the planned route 311 changes to display the caution sections, which are one piece of additional information.

[0103] The display manner of the caution section and the sections other than the caution section is not particularly limited as long as they can be distinguished from each other. For example, the caution section and the sections other than the caution section may be distinguished by color, line type, etc.

[0104] The message 312 notifies the user of a cautionary section of the planned route 311. Specifically, the message indicates that the planned route 311 of the drone 11 is approaching an object (building 301) within a predetermined distance range. The message also indicates a confirmation of whether or not to modify the planned route 311 so that the drone 11 does not approach the object (building 301) more than a predetermined distance.

[0105] This allows the user to recognize in advance the risk of the drone 11 colliding with or coming into contact with surrounding objects.

[0106] The imaging range 313 and the imaging range 314 indicate the range that is planned to be photographed by the drone 11 in the execution plan. The imaging range 313 and the imaging range 314 are displayed in different display modes based on the imaging state of the drone 11, for example, the focus state. The imaging range 313 indicates the range that is estimated to be in focus within the range that is planned to be photographed by the drone 11. The imaging range 314 indicates the range that is estimated to be out of focus within the range that is planned to be photographed by the drone 11.

[0107] The display manner of the imaging range 313 and the imaging range 314 is not particularly limited as long as they can be distinguished from each other. For example, the imaging range 313 and the imaging range 314 can be distinguished from each other by color, pattern, or the like.

[0108] This allows the user to know in advance whether or not the subject can be photographed as intended. Specifically, for example, the user can know in advance the photographing range of the subject, and the range in which the subject can and cannot be photographed in detail.

[0109] For example, the display mode of the shooting range may be changed based on the shooting direction of the subject. For example, the shooting range 313 may indicate a range within a range of 90 degrees ±α with respect to the optical axis of the shooting device (camera) of the drone 11, and the shooting range 314 may indicate a range outside the range of 90 degrees ±α with respect to the optical axis of the camera of the drone 11.

[0110] This allows the user to recognize in advance the shooting range of the subject and whether or not the subject can be photographed from a position directly facing the subject.

[0111] The menu 315 is operated when selecting and displaying only one of "shooting range" and "proximity (information)." For example, when "shooting range" is selected, the planned route 311, message 312, and menu 315 disappear from the display screen of FIG. 7, leaving the shooting range 313 and shooting range 314, as shown in FIG. 8. On the other hand, when "proximity" is selected, the message 312, shooting range 313, shooting range 314, and menu 315 disappear from the display screen of FIG. 7, leaving only the planned route 311, as shown in FIG. 9.

[0112] This improves the visibility of the additional information, allowing the user to select and check only the information that is necessary.

[0113] 10, similar to the example of Fig. 7, a 3D model of the vicinity of the building 301 is displayed as area information, and a planned route 331 is superimposed on the area information. In addition, photographing points 332a to 332c and photographing ranges 333a to 333c are superimposed on the area information as additional information.

[0114] The planned route 311 is shown by a line as the route that the drone 11 is scheduled to fly in the execution plan.

[0115] The photographing points 332a to 332c are indicated by circles as points where the drone 11 is scheduled to perform photographing on the planned route 331 in the execution plan.

[0116] Shooting ranges 333a to 333c indicate the ranges estimated to be photographed by drone 11 at shooting points 332a to 332c, respectively. Shooting ranges 333a and 333c partially overlap, and the overlapping and non-overlapping portions are displayed in different ways. For example, the color or pattern density of the overlapping portions is darker than the color or pattern density of the non-overlapping portions.

[0117] This allows the user to know in advance whether the subject can be photographed as intended. Specifically, for example, the user can know in advance whether the photographing range covers the entire subject, whether the photographing range overlaps as intended, etc.

[0118] Hereinafter, when there is no need to distinguish between the imaging points 332a to 332c, they will simply be referred to as imaging points 332. Hereinafter, when there is no need to distinguish between the imaging ranges 333a to 333c, they will simply be referred to as imaging ranges 333.

[0119] For example, the user can select one of the photography locations 332 to display only the photography range 333 corresponding to the selected photography location 332. For example, as shown in Fig. 11, when photography location 332b is selected, photography ranges 333a and 333c disappear, and only photography range 333b corresponding to photography location 332b is displayed.

[0120] This allows the user to individually recognize the photographing range 333 from each photographing point 332.

[0121] 12 , the user can display a planned route 331, photography locations 332, and photography range 333 within a period specified on a timeline 351. The timeline 351 indicates the time elapsed from the flight start time of the drone 11. Then, the section of the planned route 331 along which the drone 11 is scheduled to fly within the period specified on the timeline 351 (hereinafter referred to as the designated section), photography locations 332 within the designated section, and photography range 333 corresponding to the photography locations 332 within the designated section are displayed.

[0122] For example, in the example of FIG. 12, only the photography points 332a and 332b included in the specified section, and the photography ranges 333a and 333b corresponding to the photography points 332a and 332b are superimposed on the area information.

[0123] This allows the user to easily check the shooting status within a desired period.

[0124] 13, for example, an estimated resolution (resolving power) may be displayed for an imaging range 333. Specifically, in the example of Fig. 13, a mesh pattern indicating the resolution is displayed superimposed on an imaging range 333b. For example, the higher the resolution, the finer the mesh, and the lower the resolution, the coarser the mesh.

[0125] This allows the user to intuitively determine whether the subject will be photographed with the expected level of detail.

[0126] The resolution of the imaging range may be indicated by a pattern other than a mesh pattern.

[0127] FIG. 14 shows an example of a display of an execution plan when a mission to photograph the ground from the sky is given.

[0128] 14, an image or a two-dimensional model including the target area is displayed as area information, and a planned route 401 is superimposed on the area information. In addition, photographing points 402a to 402i and photographing ranges 403g to 403i are superimposed on the area information as additional information.

[0129] The planned route 401 is shown by a line as the route that the drone 11 is scheduled to fly in the execution plan.

[0130] The photographing points 402a to 402i are indicated by circles as points where the drone 11 is scheduled to perform photographing on the planned route 401 in the execution plan.

[0131] Photographing ranges 403g to 403i indicate the ranges estimated to be photographed by the drone 11 at photographing points 402g to 402i, respectively. Photographing points 402g to 402i partially overlap, and the overlapping portions and non-overlapping portions are displayed in different ways. For example, the color or pattern density of the overlapping portions is darker than the color or pattern density of the non-overlapping portions.

[0132] 14, for ease of understanding, the imaging ranges 403a to 403f corresponding to the imaging points 402a to 402f are not shown. Also, only the auxiliary line connecting the imaging point 402i to the imaging range 403i is shown, and the auxiliary lines connecting the other imaging points to the imaging ranges are not shown.

[0133] This allows the user to know in advance whether the subject can be photographed as intended. Specifically, for example, the user can know in advance whether the photographing range covers the entire subject, whether the photographing range overlaps as intended, etc.

[0134] FIG. 15 shows an example of a display of an execution plan when a mission is given to photograph an area including a building 501 on the ground from the sky.

[0135] In this example, an image or two-dimensional model of the area including the target area is displayed as area information, and a planned route 511 is superimposed on the area information. In addition, a dangerous area 512 is superimposed on the area information as additional information.

[0136] The planned route 511 shows the flight path of the drone 11 in the execution plan using a line. Within the planned route 511, sections requiring caution are displayed in a different manner from other sections. In this example, sections of the planned route 511 where the distance to the building 501 is less than the approach judgment threshold are displayed as caution sections using dotted lines. In this way, the change in the display manner of the planned route 511 displays the caution sections, which are one piece of additional information.

[0137] The display manner of the caution section and the section other than the caution section is not particularly limited as long as it is possible to distinguish them from each other.

[0138] In addition, in the caution section, an area where there is a risk of the drone 11 colliding with or coming into contact with the building 501 is displayed as a rectangular danger area 512.

[0139] In step S12, the execution plan creation unit 112 of the user terminal 13 determines whether or not to modify the execution plan.

[0140] For example, the user refers to the execution plan displayed on the output unit 103 and determines whether to finalize the execution plan. Then, the user inputs information indicating whether to finalize or modify the execution plan via the input unit 101. The input unit 101 supplies input information including the input information to the control unit 102.

[0141] In response to this, the execution plan creation unit 112 determines whether or not to modify the execution plan based on the input information. If it is determined that the execution plan should be modified, the process returns to step S4.

[0142] Thereafter, the processes of steps S4 to S12 are repeatedly executed until it is determined in step S12 that the execution plan is to be finalized, whereby the user modifies the mission and the execution plan is modified accordingly.

[0143] The user may also directly modify the execution plan.

[0144] On the other hand, if it is determined in step S12 that the execution plan is to be finalized, the process proceeds to step S13.

[0145] In step S13, the execution plan creation unit 112 of the user terminal 13 transmits the execution plan information. For example, the execution plan creation unit 112 transmits the execution plan information to the drone 11 via the communication unit 104 and the network 21, or via the communication unit 104 and the controller 12.

[0146] Thereafter, the execution plan creation process ends.

[0147] Thereafter, the drone 11 autonomously executes the mission given by the user based on the execution plan information.

[0148] In this way, the user can appropriately create an execution plan for carrying out the desired mission while checking whether the intended results will be obtained before flying the drone 11.

[0149] For example, the user can easily check the flight path of the drone 11 and the shooting status of the drone 11. For example, the user can check whether the flight path of the drone 11 maintains a safe distance from the subject. For example, the user can check whether the drone 11 can capture all necessary parts of the subject, whether it can capture images with the appropriate focus, whether it can capture images from the appropriate direction, whether it can capture images with the appropriate resolution, etc. Then, based on the results of the check, the user can modify the mission and execution plan as necessary.

[0150] This can avoid mission reruns, saving time and reducing risk during mission execution.

[0151] <<2. Modifications>> Modifications of the above-described embodiments of the present technology will now be described.

[0152] <Modifications of Processing Allocation> The above-described processing allocation can be changed as appropriate.

[0153] For example, the user terminal 13 may execute part or all of the processes of the server 14 described above, or the server 14 may execute part or all of the processes of the user terminal 13 described above. For example, the user terminal 13 may have a function equivalent to the additional information generation unit 163 of the server 14, and generate additional information based on area information, device information, and execution plan information. For example, the server 14 may have a function equivalent to the execution plan creation unit 112 of the user terminal 13, and create an execution plan based on mission information related to a mission given by the user.

[0154] For example, the drone 11 or the controller 12 may perform some or all of the processing of the user terminal 13 described above.

[0155] <Modifications on the method of displaying the execution plan> For example, when the drone 11 is flying according to the execution plan, the actual flight path of the drone may be displayed superimposed on the planned path.

[0156] 16 shows an example of a display screen when the drone 11 is flying. In this example, a flight path 531 and a current position 532 are newly superimposed on the display screen of FIG.

[0157] The flight path 531 shows the actual path of the drone 11 during flight. The flight path 531 is displayed in a different display mode from the planned path 511 so as to be distinguishable from the planned path 511. Furthermore, within the flight path 531, a section where the distance between the drone 11 and the building 501 actually falls below the approach determination threshold (hereinafter referred to as a "approach section") is displayed in a different display mode from a section that is not a approach section (hereinafter referred to as a "non-approach section").

[0158] The display manner of the approach section and non-approach section of the planned route 511 and the flight route 531 is not particularly limited as long as they can be distinguished from each other.

[0159] The current position 532 indicates the current position of the drone 11.

[0160] This allows the user to, for example, check whether the drone 11 is flying as planned, whether it is flying safely, etc. Furthermore, the user can adjust the flight path of the drone 11 by remotely controlling the drone 11 as necessary.

[0161] Also, for example, after the drone 11 flies according to the execution plan, the actual flight path of the drone may be displayed superimposed on the planned path.

[0162] Fig. 17 shows an example of a display screen after the drone 11 has flown. In this example, a flight path 531 is newly superimposed on the display screen of Fig. 15. In addition, since the flight has ended and no particular attention is required, the dangerous area 512 has been erased.

[0163] The display mode of the flight path 531 is the same as the display mode of the flight path 531 in FIG.

[0164] This allows, for example, the user to check after the drone 11 has flown whether the drone 11 flew as planned.

[0165] <Other Modifications> For example, the user terminal 13 or another device may generate a mission instead of the user. For example, the user terminal 13 or another device may automatically generate a new mission that needs to be executed based on the execution history of past missions.

[0166] This technology can be applied to autonomous flying objects other than drones, such as flying cars that can autonomously fly and execute missions.

[0167] In addition to autonomous aerial vehicles, the present technology can be applied to any autonomous mobile object that can move autonomously according to a mission and execute the mission. For example, the present technology can be applied to a self-driving vehicle that executes a mission autonomously.

[0168] <<3. Others>> <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.

[0169] FIG. 18 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0170] In the computer 1000 , a CPU (Central Processing Unit) 1001 , a ROM (Read Only Memory) 1002 , and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004 .

[0171] An input / output interface 1005 is further connected to the bus 1004. An input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010 are connected to the input / output interface 1005.

[0172] The input unit 1006 includes input switches, buttons, a microphone, an image sensor, etc. The output unit 1007 includes a display, a speaker, etc. The storage unit 1008 includes a hard disk, a non-volatile memory, etc. The communication unit 1009 includes a network interface, etc. The drive 1010 drives removable media 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0173] In the computer 1000 configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program recorded in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0174] The program executed by the computer 1000 (CPU 1001) can be provided by being recorded on a removable medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0175] In the computer 1000, the program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting the removable medium 1011 into the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be installed in the ROM 1002 or the storage unit 1008 in advance.

[0176] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0177] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0178] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0179] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0180] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0181] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0182] <Examples of Combinations of Configurations> The present technology can also have the following configurations.

[0183] (1) An information processing device comprising: an output control unit that controls display of execution plan information related to an execution plan for an autonomous moving body to execute a mission, area information related to an execution area in which the mission will be executed, and additional information applicable to determining the validity of the execution plan. (2) The information processing device described in (1), wherein the mission includes a mission to photograph a subject, and the additional information includes shooting state information related to a shooting state of the subject by a photographing device provided in the autonomous moving body. (3) The information processing device described in (2), wherein the shooting state information includes at least one of a shooting range by the photographing device, a shooting direction of the subject, a focus state for the subject, and a resolution of the subject. (4) The information processing device described in (3), wherein the output control unit controls to display the shooting range in a different display mode based on at least one of the shooting direction and the focus state. (5) The information processing device described in (3) or (4), wherein the output control unit controls to display an overlapping portion of the shooting range in a different display mode from other portions. (6) The information processing device according to any one of (3) to (5), wherein the output control unit controls to display a pattern indicating the resolution of the subject to be photographed so as to be superimposed on the photographing range. (7) The information processing device according to any one of (2) to (6), further comprising an additional information generation unit that generates the additional information including the photographing state information based on the area information, the execution plan information, and device information related to the photographing device. (8) The execution plan information includes a planned route that is a planned movement route of the autonomous moving body, and the additional information includes information on a cautionary section on the planned route that poses a risk of collision or contact with surrounding objects. (9) The information processing device according to (8), wherein the output control unit controls to display the planned route in different display modes for the cautionary section and other sections. (10) The information processing device according to (8) or (9), wherein the output control unit controls to display an area in the cautionary section that poses a risk of collision or contact for the autonomous moving body.(11) The information processing device according to any of (8) to (10), wherein the output control unit controls to display the planned route and the actual movement route of the autonomous moving body in an overlapping manner at least one of during and after movement of the autonomous moving body. (12) The information processing device according to any of (8) to (11), wherein the caution section is set based on a distance between the planned route and a surrounding object. (13) The information processing device according to (12), wherein the caution section is set further based on at least one of operation information of the autonomous moving body and environmental information about the surroundings of the planned route. (14) The information processing device according to any of (8) to (13), further comprising: an additional information generation unit that generates the additional information including the caution section based on the area information and the planned route. (15) The information processing device according to any of (1) to (14), further comprising: an execution plan creation unit that creates the execution plan based on mission information related to the mission and the area information. (16) The information processing device according to (1), further comprising an additional information generation unit that generates the additional information based on the execution plan information and the area information. (17) The information processing device according to any of (1) to (16), wherein the area information includes at least one of map information, an image, a two-dimensional model, and a three-dimensional model of the vicinity of the execution area. (18) The information processing device according to any of (1) to (17), wherein the autonomous moving body is an autonomous air vehicle. (19) An information processing method, wherein an information processing device controls display of execution plan information related to an execution plan for an autonomous moving body to execute a mission, area information related to an execution area in which the mission is to be executed, and additional information applicable to determining the validity of the execution plan.(20) An information processing system comprising: an execution plan creation unit that creates an execution plan for an autonomous moving body to execute a mission based on mission information regarding a mission to be executed by the autonomous moving body and area information regarding an execution area in which the mission will be executed; an additional information generation unit that generates additional information applicable to determining the validity of the execution plan based on the execution plan information regarding the execution plan and the area information; and an output control unit that controls display of the execution plan information, the area information, and the additional information, wherein the execution plan creation unit, the additional information generation unit, and the output control unit are each disposed in any of two or more information processing devices.

[0184] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0185] REFERENCE SIGNS LIST 1 Information processing system, 11 Drone, 12 Controller, 13 User terminal, 14 Server, 102 Control unit, 103 Output unit, 104 Communication unit, 111 Information acquisition unit, 112 Execution plan creation unit, 113 Output control unit, 114 Remote control unit, 151 Control unit, 152 Communication unit, 161 Information acquisition unit, 162 Area information generation unit, 163 Additional information generation unit

Claims

1. An information processing device having an output control unit that controls the display of execution plan information regarding an execution plan for an autonomous moving body to execute a mission, area information regarding an execution area in which the mission will be executed, and additional information applicable to determining the validity of the execution plan.

2. The information processing device according to claim 1, wherein the mission includes a mission of photographing a subject, and the additional information includes photographing condition information regarding a photographing condition of the subject by a photographing device provided on the autonomous moving body.

3. The information processing device according to claim 2, wherein the shooting condition information includes at least one of the shooting range of the shooting device, the shooting direction of the subject, the focus state of the subject, and the resolution of the subject.

4. The information processing device according to claim 3, wherein the output control unit controls the imaging range to be displayed in a different display mode based on at least one of the imaging direction and the focus state.

5. The information processing device according to claim 3, wherein the output control unit controls so that the portion where the shooting ranges overlap is displayed in a display mode different from that of the other portion.

6. The information processing device according to claim 3, wherein the output control unit controls so as to display a pattern indicating the resolution of the subject to be photographed in a superimposed manner on the photographing range.

7. The information processing device according to claim 2, further comprising an additional information generating unit that generates the additional information including the shooting condition information based on the area information, the execution plan information, and device information related to the shooting equipment.

8. The information processing device according to claim 1, wherein the execution plan information includes a planned route, which is a planned movement route of the autonomous moving body, and the additional information includes information regarding cautionary sections of the planned route where there is a risk of collision or contact with surrounding objects.

9. The information processing device according to claim 8, wherein the output control unit controls the planned route to be displayed in a different display mode for the caution-required section and other sections.

10. The information processing device according to claim 8, wherein the output control unit controls the display of areas in the caution section where there is a risk of collision or contact with the autonomous moving body.

11. The information processing device according to claim 8, wherein the output control unit controls to display the planned route and the actual movement route of the autonomous moving body in an overlapping manner at least one of during and after the movement of the autonomous moving body.

12. The information processing device according to claim 8, wherein the caution section is set based on a distance between the planned route and surrounding objects.

13. The information processing device according to claim 12, wherein the caution required section is set further based on at least one of operation information of the autonomous moving body and environmental information surrounding the planned route.

14. The information processing device according to claim 8, further comprising an additional information generating unit that generates the additional information including the caution-required section based on the area information and the planned route.

15. The information processing device according to claim 1, further comprising an execution plan creation unit that creates the execution plan based on mission information related to the mission and the area information.

16. The information processing device according to claim 1, further comprising an additional information generating unit that generates the additional information based on the execution plan information and the area information.

17. The information processing device according to claim 1, wherein the area information includes at least one of map information, an image, a two-dimensional model, and a three-dimensional model of the vicinity of the execution area.

18. The information processing device according to claim 1, wherein the autonomous moving body is an autonomous flying body.

19. An information processing method in which an information processing device controls the display of execution plan information regarding an execution plan for an autonomous moving body to execute a mission, area information regarding an execution area in which the mission will be executed, and additional information applicable to determining the validity of the execution plan.

20. An information processing system comprising: an execution plan creation unit that creates an execution plan for an autonomous mobile body to execute a mission based on mission information regarding a mission to be executed by the autonomous mobile body and area information regarding an execution area in which the mission will be executed; an additional information generation unit that generates additional information applicable to determining the validity of the execution plan based on the execution plan information regarding the execution plan and the area information; and an output control unit that controls the display of the execution plan information, the area information, and the additional information, wherein the execution plan creation unit, the additional information generation unit, and the output control unit are each disposed in any of two or more information processing devices.

Images