Information processing method, information processing system, and program

Abstract

To enable information for moving a moving object to be generated intuitively.SOLUTION: An information processing system is provided with: a display control unit for controlling display, on a display screen, of a virtual object based on an object being present in a real space; and a movement information generation unit for generating movement information for controlling the movement of a moving object. Thus, the information for moving the moving object can be generated intuitively.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, mobile objects such as drones that are controlled using a control device, etc., have been used. For example, a drone equipped with a camera is made to capture images of scenery from the sky, and the captured images are utilized.

[0003] For example, Patent Document 1 describes a technique for efficiently transferring images by switching from an image capturing mode to an image transfer mode when a predetermined mode switching condition occurs. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-16869 Summary of the Invention [Problem to be solved by the invention]

[0005] As a method for controlling the movement of a mobile object such as a drone, in addition to a method in which a user controls the movement of the mobile object using a control device, a method in which a route for the mobile object to travel is set in advance and the mobile object moves along the set route can be considered. In this case, for example, a method in which the route for the mobile object to travel can be set on a map can be considered.

[0006] However, when a moving object moves three-dimensionally, it is difficult to intuitively set the movement of the moving object using a method of setting the path of the moving object on a two-dimensional map. The technology described in Patent Document 1 is not intended to intuitively generate movement information for controlling the movement of a moving object such as a drone.

[0007] Therefore, the present disclosure proposes a new and improved information processing device, information processing method, and program that are capable of intuitively generating information for moving a moving object. [Means for solving the problem]

[0008] According to the present disclosure, an information processing device is provided that includes a display control unit that controls the display on a display screen of a virtual object based on an object that exists in real space, and a movement information generation unit that generates movement information for controlling the movement of a moving body.

[0009] The present disclosure also provides an information processing method including a processor controlling the display of a virtual object based on an object existing in real space on a display screen, and generating movement information for controlling the movement of a moving body.

[0010] Furthermore, according to the present disclosure, a program is provided for enabling a computer to realize a function of controlling the display of virtual objects based on objects existing in real space on a display screen, and a function of generating movement information for controlling the movement of a moving body. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating a configuration of an information processing system according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a functional block diagram illustrating a configuration of a user terminal according to an embodiment of the present disclosure. [Figure 3] FIG. 2 is a functional block diagram showing the configuration of a processing unit. [Figure 4] FIG. 10 is a diagram showing a user operating a mobile object to capture images of a tower and a forest. [Figure 5] FIG. 10 is a diagram showing a virtual object generated based on an image of a tower. [Figure 6] FIG. 10 is a diagram showing a virtual object generated based on an image of a forest. [Figure 7] FIG. 2 is a diagram showing a route traveled by a moving object. [Figure 8] FIG. 10 is a diagram showing how the surface of a desk in real space is detected by the user terminal. [Figure 9] FIG. 10 is a diagram showing a state in which a Waypoint is selected based on a user operation. [Figure 10] FIG. 10 is a diagram showing a state in which a Waypoint is selected based on a user operation. [Figure 11] FIG. 10 is a diagram showing how the position of a Waypoint is adjusted based on a user operation. [Figure 12] FIG. 10 is a diagram showing how the position of a Waypoint is adjusted based on a user operation. [Figure 13] FIG. 10 is a diagram showing a state in which a new route for a moving object is set based on a user operation. [Figure 14] FIG. 10 is a diagram showing a state in which a new route for a moving object is set based on a user operation. [Figure 15] FIG. 10 is a diagram showing how the position of an imaging unit included in a user terminal is set as a Waypoint. [Figure 16] FIG. 10 is a diagram showing a display screen when the position of an imaging unit included in a user terminal is set as a Waypoint. [Figure 17] FIG. 10 is a diagram showing how a position a predetermined distance away from the user terminal is set as a Waypoint. [Figure 18] FIG. 10 is a diagram showing a display screen when a position a predetermined distance away from the user terminal is set as a Waypoint. [Figure 19] FIG. 10 is a diagram showing how a Waypoint is set using a designation stick. [Figure 20] FIG. 10 is a diagram showing the display screen when a Waypoint is set using the designation stick. [Figure 21] FIG. 10 is a diagram showing how the orientation of an imaging device of a moving object is set by moving the orientation of a user terminal. [Figure 22]10A and 10B are diagrams illustrating how the angle of view of an imaging device of a moving object is set by performing a pinch operation on the display screen of a user terminal. [Figure 23] FIG. 10 is a diagram showing a display screen that displays a simulation result of the movement of a moving object. [Figure 24] FIG. 10 is a diagram showing a display screen that displays a simulation result of an image captured by an imaging device provided in a moving object. [Figure 25] FIG. 10 is a flowchart showing a manual imaging method. [Figure 26] FIG. 10 is a flowchart showing a method for automatically flying a moving object and causing an imaging device to capture video. [Figure 27] FIG. 10 is a flowchart illustrating a procedure for generating a virtual object. [Figure 28] FIG. 10 is a flowchart showing the procedure for capturing an image based on the generated movement information and imaging information. [Figure 29] FIG. 10 is a diagram illustrating a display process performed by an information processing device. [Figure 30] FIG. 2 is a functional block diagram illustrating an example of a hardware configuration of a user terminal that configures an information processing system according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, multiple components having substantially the same functional configuration may be distinguished by adding different letters after the same reference numeral. For example, multiple components having substantially the same functional configuration may be distinguished as needed, such as user terminal 10a and user terminal 10b. However, if there is no need to particularly distinguish between multiple components having substantially the same functional configuration, only the same reference numeral will be used. For example, if there is no need to particularly distinguish between user terminal 10a and user terminal 10b, they will simply be referred to as user terminal 10.

[0013] The explanation will be given in the following order. 1. Configuration 1.1. Information Processing System Configuration 1.2. User terminal configuration 2. Virtual object generation 3. Operation example 3.1. Generation of movement information 3.2. Generation of imaging information 3.3.Simulation of the movement of moving objects 4. Imaging method 4.1.Manual imaging method 4.2. Automatic flight imaging method 4.3. Imaging method using a map displayed on the device screen 4.4. Imaging method according to the present disclosure 5.Effects 6. Hardware Configuration 7. Supplementary Information

[0014] <1. Configuration> <<1.1. Information Processing System Configuration>> First, a configuration of an information processing system 1 according to an embodiment of the present disclosure will be described with reference to Fig. 1. Fig. 1 is a diagram showing the configuration of the information processing system 1 according to an embodiment of the present disclosure. The information processing system 1 includes a user terminal 10 and a mobile object 20. The user terminal 10 and the mobile object 20 are connected to each other so that they can communicate with each other.

[0015] The user terminal 10 may be, for example, a smartphone or a tablet terminal. The user terminal 10 generates movement information for controlling the movement of the moving object 20 in response to a user operation, and transmits the movement information to the moving object 20. The user terminal 10 can also display virtual objects, which will be described later, in response to a user operation.

[0016] The mobile object 20 is a device that moves based on movement information generated by the user terminal 10. Here, the mobile object 20 can be any type of mobile device, but the following description will be given assuming that the mobile object 20 is a drone. The mobile object 20 may also be equipped with an imaging device for capturing images of scenery.

[0017] <<1.2. User Device Configuration>> The configuration of the user terminal 10 according to an embodiment of the present disclosure will be described with reference to Fig. 2. Fig. 2 is a functional block diagram showing the configuration of the user terminal 10 according to an embodiment of the present disclosure.

[0018] The user terminal 10 has a function of acquiring image information, sensor information, information based on user operations, etc., and outputting the results of various processes performed on the acquired information. The functions of the user terminal 10 are realized by cooperation of the information processing device 100, imaging unit (first imaging device) 110, sensor unit 120, input unit 130, and display unit 175 provided in the user terminal 10.

[0019] The imaging unit 110 may be any of various known imaging devices that capture images. The imaging unit 110 has any of various known imaging elements, such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. In addition to these imaging elements, the imaging unit 110 may also have various components, such as a lens that forms an image of a subject on the imaging element and a light source that irradiates the subject with illumination light. The imaging unit 110 transmits image information obtained by capturing an image to the information processing device 100.

[0020] The sensor unit 120 includes at least one of various known sensors, such as a distance measurement sensor or an IMU (Inertial Measurement Unit). The distance measurement sensor may be, for example, a stereo camera or a ToF (Time of Flight) sensor. The distance measurement sensor detects distance information relating to, for example, the distance between the user terminal 10 and an object present around it, and transmits the detected distance information to the information processing device 100. The IMU includes, for example, at least one of an acceleration sensor, a gyro sensor, and a magnetic sensor. The IMU transmits the detected information to the information processing device 100 as IMU information.

[0021] The input unit 130 has a function of generating input information based on an operation by a user. The input unit 130 may be, for example, a touch panel. The input unit 130 generates the input information based on various operations by the user, such as a touch operation, a drag operation, a pinch-out operation, or a pinch-in operation. The input unit 130 transmits the generated input information to the acquisition unit 140.

[0022] The information processing device 100 has a function of performing various processes based on the acquired information and, based on the results of the processes, controlling the display on the display unit 175. The functions of the information processing device 100 are realized by the acquisition unit 140, the processing unit 150, the display control unit 170, the storage unit 180, and the communication control unit 190 working together.

[0023] The acquisition unit 140 acquires information input from at least one of the imaging unit 110, the sensor unit 120, and the input unit 130. The acquisition unit 140 transmits the acquired information to the processing unit 150.

[0024] The processing unit 150 has a function of performing various processes based on the information transmitted from the acquisition unit 140. For example, the processing unit 150 has a function of generating information for controlling the moving object 20 based on the information transmitted from the acquisition unit 140. The processing unit 150 also generates information related to the content to be displayed on the display screen of the display unit 175. The detailed configuration and functions of the processing unit 150 will be described later with reference to FIG. 3. The processing unit 150 transmits the generated information to the display control unit 170, the storage unit 180, or the communication control unit 190.

[0025] The display control unit 170 has a function of controlling the display on the display screen of the display unit 175. For example, based on information transmitted from the processing unit 150, the display control unit 170 controls the display of virtual objects based on objects existing in real space on the display screen of the display unit 175.

[0026] The display unit 175 is a display device having a function of displaying various known images. In this embodiment, the display unit 175 and the above-mentioned input unit 130 are integrated and configured as a touch panel. As will be described later, the user can cause the information processing device 100 to generate movement information for controlling the movement of the mobile object 20 in the user terminal 10 by performing a predetermined operation while referring to the display screen of the display unit 175.

[0027] The storage unit 180 has a function of storing various types of information, such as information generated or acquired by the information processing device 100. For example, the storage unit 180 may store information related to virtual objects generated in advance. Note that a method for generating virtual objects will be described later. The storage unit 180 may also store movement information for controlling the movement of the moving object 20. More specifically, the storage unit 180 may store information (waypoint information) related to specific points (also referred to as "waypoints") included in the route of the moving object 20. The route of the moving object 20 may be formed by connecting multiple waypoints. The storage unit 180 may also store movement information or imaging information generated by the processing unit 150. The information stored in the storage unit 180 is referenced by the processing unit 150, the display control unit 170, or the communication control unit 190 as necessary.

[0028] The communication control unit 190 has a function of controlling the transmission of various types of information generated by the processing unit 150. The communication control unit 190 controls the transmission of movement information or imaging information generated by the processing unit 150. The movement information or imaging information is transmitted to the moving object 20. The moving object 20 can move or capture images based on the transmitted information.

[0029] Next, the processing unit 150 included in the information processing device 100 will be described in more detail with reference to Fig. 3. Fig. 3 is a functional block diagram showing the configuration of the processing unit 150. As shown in Fig. 3, the processing unit 150 includes a detection unit 151, a self-position calculation unit 154, a virtual object calculation unit 155, a generation unit 156, and a prediction unit 160.

[0030] The detection unit 151 has a function of performing various detections based on information transmitted from the acquisition unit 140. The functions of the detection unit 151 are realized by the plane detection unit 152 and the object detection unit 153. The plane detection unit 152 has a function of detecting a plane included in an image based on image information, distance information, etc. The object detection unit 153 has a function of detecting a predetermined object included in an image based on image information, distance information, etc. The detection unit 151 transmits the detection result to the self-position calculation unit 154.

[0031] The self-position calculation unit 154 has a function of calculating the self-position of the user terminal 10. Here, the self-position of the user terminal 10 includes not only the position where the user terminal 10 is present but also the orientation of the user terminal 10. Specifically, the self-position calculation unit 154 receives image information, distance information, and IMU information as input, and calculates the position or orientation of the user terminal 10 relative to the environment or objects around the user terminal 10 using SLAM (Simultaneous Localization And Mapping) technology. At this time, the self-position calculation unit 154 may determine the origin or scale in SLAM based on object information, plane information, or the like. The self-position calculation unit 154 transmits the calculation results to the virtual object calculation unit 155 and the generation unit 156.

[0032] The virtual object calculation unit 155 generates information about a virtual object to be placed on the display screen. More specifically, the virtual object calculation unit 155 calculates placement information (information about the position, direction, etc.) or scale information about the virtual object to be placed on the display screen of the display unit 175 based on the self-position of the user terminal 10 calculated by the self-position calculation unit 154, the detection result by the detection unit 151, input information input to the input unit 130, information about the virtual object stored in the storage unit 180, etc. Here, the virtual object calculation unit 155 calculates the scale of the virtual object based on the scale of real space. More specifically, the virtual object calculation unit 155 determines the scale of the virtual object displayed on the display screen by appropriately enlarging or reducing the scale of the real object that is the basis of the virtual object, and generates scale information. The virtual object calculation unit 155 transmits the calculation result to the movement information generation unit 157, which will be described later.

[0033] The generation unit 156 has a function of generating various types of information for controlling the moving object 20. More specifically, the generation unit 156 generates information for controlling the movement of the moving object 20, the operation of the imaging device (second imaging device) included in the moving object 20, and the display of the display unit 175. The functions of the generation unit 156 are realized by a movement information generation unit 157, an imaging information generation unit 158, and a display information generation unit 159 included in the generation unit 156.

[0034] The movement information generation unit 157 generates movement information associated with a virtual object for controlling the movement of the moving body 20. Specifically, the movement information generation unit 157 generates the position and direction of a waypoint as movement information based on the self-position of the user terminal 10, input information to the input unit 130, placement information of the virtual object, scale information, and waypoint information. For example, the movement information generation unit 157 may generate the path of the moving body 20 as movement information. In this case, the movement information generation unit 157 may generate movement information of a scale corresponding to the scale information of the virtual object, or may generate movement information by adjusting the movement information to the scale of the real space. Furthermore, the movement information generation unit 157 can also modify the movement information and generate new movement information based on operations by the user on the input unit 130, etc. Specific user operations will be described later. The movement information generation unit 157 transmits the generated movement information to the display information generation unit 159, the prediction unit 160, and the storage unit 180.

[0035] The imaging information generation unit 158 ​​has a function of generating imaging information for controlling the range of imaging by the imaging device provided in the moving object 20, based on a user's operation. More specifically, the imaging information generation unit 158 ​​generates imaging information related to the direction or angle of view of the imaging device, etc., based on input information, etc., generated by the input unit 130. The imaging information generation unit 158 ​​transmits the generated imaging information to the prediction unit 160 and the storage unit 180.

[0036] The display information generation unit 159 has a function of generating display information related to the content to be displayed on the display unit 175. More specifically, the display information generation unit 159 generates CG (Computer Graphics) to be displayed on the display unit 175 as display information based on the position of the imaging unit 110, placement information of virtual objects, scale information, Waypoint information, a prediction result by the prediction unit 160 (described later), and the like. The display information generation unit 159 can generate an image showing the movement of the moving object 20 as display information. The display information generation unit 159 can also generate display information for displaying a simulation image captured by an imaging device provided in the moving object 20. The display information generation unit 159 transmits the generated display information to the display control unit 170.

[0037] The prediction unit 160 has a function of predicting the movement of the moving object 20. More specifically, the prediction unit 160 can predict the movement of the moving object 20 and the movement of the imaging device provided in the moving object 20. The function of the prediction unit 160 is realized by the movement prediction unit 161 and the imaging prediction unit 162. The prediction unit 160 transmits the predicted result to the display information generation unit 159.

[0038] The movement prediction unit 161 has a function of predicting the movement of the moving object 20 based on the movement information. For example, the movement prediction unit 161 can predict the path of the moving object 20. Furthermore, the imaging prediction unit 162 predicts an image to be captured by an imaging device provided in the moving object 20 based on the movement information and imaging information. More specifically, the imaging prediction unit 162 predicts an image to be captured by the imaging device based on the path that the imaging device will take, the attitude of the imaging device, and the like.

[0039] <2. Virtual object generation> In this embodiment, a virtual object is displayed on the display unit 175, and the user can cause the information processing device 100 to generate movement information, imaging information, or the like by performing a predetermined operation while viewing the display. Here, an example of a method for generating the virtual object will be described. Note that the method for generating the virtual object is not limited to the method described below, and the virtual object may be generated by any method.

[0040] A method for generating a virtual object will be described with reference to Figs. 4 to 7. Fig. 4 is a diagram showing a user U1 operating a moving object 20 to capture an image of a tower 420 and a forest 430. Fig. 5 is a diagram showing a virtual object 422 generated based on the captured image of the tower 420. Fig. 6 is a diagram showing a virtual object 432 generated based on the captured image of the forest 430. Fig. 7 is a diagram showing a route 402 traveled by the moving object 20.

[0041] First, a method for generating a virtual object according to this embodiment will be briefly described. Here, it is assumed that user U1 wants to have the imaging device 206 capture an image including a tower 420. In this embodiment, first, user U1 uses the control device 401 to operate the moving object 20, and causes the imaging device 206 to capture images of the tower 420 existing in real space and the forest 430 existing around the tower 420 in advance. Next, a three-dimensional virtual object is generated based on the captured images and various CG technologies. In this embodiment, waypoints are further set on the route traveled by the moving object 20, and movement information is generated. Hereinafter, the method for generating a virtual object will be described in more detail with reference to FIGS. 4 to 7.

[0042] First, as shown in FIG. 4, user U1 uses a control device 401 to fly a moving body (drone) 20. The moving body 20 includes an airframe 202, a propeller 204, and an imaging device 206. The propeller 204 is driven to enable the moving body 20 to fly. Furthermore, the user U1 controls the propeller 204 and other components to control the direction, attitude, speed, and so on of the airframe 202. Furthermore, the imaging device 206 included in the moving body 20 captures an image of the scenery around the moving body 20. Here, an image including a tower 420 and the surrounding forest 430 shown in FIG. 4 is captured by the imaging device 206.

[0043] Based on the captured image, various known CG techniques are used to generate virtual objects of a tower 420 and a forest 430. More specifically, a virtual object 422 of the tower 420 existing in the real space shown in Fig. 5 and a virtual object 432 of the forest 430 shown in the real space shown in Fig. 6 are generated. Information related to the virtual objects generated in this manner is stored in the storage unit 180 included in the information processing device 100.

[0044] Furthermore, the user U1 may cause the moving body 20 to fly in a manner that will actually capture a desired image. More specifically, the user U1 controls the moving body 20 so that the moving body 20 passes through a route 402 that circles around the tower 420 shown in FIG. 4. The moving body 20 calculates the route 402 traveled using a sensor provided in the moving body 20, and records the calculation result on a recording medium or the like provided in the moving body 20. Here, waypoints may be set on the route 402 according to a predetermined rule. For example, waypoints may be set at predetermined distances. Furthermore, the density of waypoints may be adjusted according to the curvature of the route 402.

[0045] Fig. 7 shows an example of a route 402 on which waypoints 406 are set. Note that Fig. 7 shows 13 waypoints 406a to 406m, but this is not limiting, and 2 to 12 waypoints, or 14 or more waypoints may be set on the route 402. Note that while waypoints may be shown in the figures used in the following explanation, the number of waypoints is not limited to the number shown in those figures.

[0046] Information about the route 402 on which the waypoints are set in this way may be stored as travel information in the storage unit 180 included in the information processing device 100.

[0047] <3. Operation example> Here, a specific example of an operation performed by a user to cause the information processing device 100 to generate movement information, imaging information, or the like based on the virtual object generated as described above will be described.

[0048] <<3.1. Generation of movement information>> A case where movement information is generated based on a user's operation will be described. First, with reference to Figs. 8 to 12, a case where movement information is recorded in advance in the information processing device 100 and the movement information is modified to generate new movement information will be described. Fig. 8 is a diagram showing how the plane on the desk 500 existing in the real space is detected by the user terminal 10a. Figs. 9 and 10 are diagrams showing how the Waypoint 408a is selected based on the operation of the user U2. Furthermore, Figs. 11 and 12 are diagrams showing how the position of the Waypoint 408a is adjusted based on the operation of the user U2.

[0049] As shown in FIG. 8, it is assumed that a desk 500 exists in the real space in front of a user U2. The user U2 is holding a user terminal 10a, and a start button 602 for starting the display of a virtual object and the setting of a waypoint is displayed on a display screen 610 of the user terminal 10a. Here, the display screen 610 also functions as an input unit 130 that accepts touch operations, pinch operations, and the like from the user U2. When the user U2 touches the start button 602, the user terminal 10a detects a plane 506 on the desk 500.

[0050] Then, as shown in FIG. 9 , an image 612a of a virtual tower object 422a generated in advance and an image 614a of a virtual route 404a of the moving object 20 are displayed on the display screen 610a of the user terminal 10a. Here, the virtual route is a route in a virtual space on which the virtual object is placed. The scale of the virtual route is appropriately increased or decreased to form a real route along which the moving object 20 actually moves. Hereinafter, when there is no particular distinction between the virtual route and the real route, they are also simply referred to as a "route." At this time, the virtual object 422a is displayed on the display screen 610a as if it were placed on the desk 500. Note that, although the virtual object 422a is shown on the desk 500 in FIG. 9 , the virtual object 422a is shown here for the sake of explanation, and the virtual object 422a is not actually placed on the desk 500.

[0051] Here, the size of the image 614a of the virtual object 422a may be, for example, a size that can be placed on a desk 500 as shown in FIG. 9. Furthermore, an image 614a of the virtual route 404a displays an image 616a of a waypoint 408a for adjusting the virtual route 404a. In this embodiment, the user U2 can also adjust the size of the image 612a of the virtual object displayed on the display screen 610a (i.e., the distance from the user terminal 10a to the virtual object 422a) by performing a pinch operation or the like on the display screen 610a. At this time, the user terminal 10a may store the ratio between the size of the tower that exists in the real space and that is the basis for the virtual object 422a and the size of the virtual object 422a.

[0052] In this embodiment, the image 612a of the virtual object 422a is displayed on a plane (on the desk 500), so the user U2 feels as if the virtual object 422a is placed on the ground. This allows the user U2 to operate the user terminal 10a more intuitively.

[0053] Here, it is assumed that the moving object 20 has been flown in advance by manual control or the like, and that the virtual route 404a has been generated based on this flight. It is also assumed that waypoints have been set in advance on the virtual route 404a. Specifically, as shown in FIG. 9, 13 waypoints indicated by circles have been set on the virtual route 404a. Furthermore, one waypoint 408a of the 13 waypoints corresponds to the waypoint image 616a displayed on the display screen 610a.

[0054] The user U2 can select a waypoint to be adjusted by touching the waypoint image 616a displayed on the display screen 610a. Here, it is assumed that the user U2 has selected the waypoint 408a on the virtual route 404a. Furthermore, with the waypoint 408a selected, the user U2 can adjust the position of the waypoint 408a by performing a pinch operation, a drag operation, or the like on the display screen 610a. In this embodiment, the user U2 can also adjust the orientation of the imaging device provided in the moving object 20 by operating the display screen 610a. For example, with the waypoint 408a selected, the user U2 can specify the orientation of the imaging device of the moving object 20 when the moving object 20 passes through a position corresponding to the waypoint 408a by performing a pinch operation, a drag operation, or the like on the display screen 610a.

[0055] Next, the user U2 moves the position of the user terminal 10a with the waypoint 408a selected. For example, as shown in FIGS. 11 and 12, the user U2 pulls the user terminal 10a toward the user U2. As a result, the position of the selected waypoint 408a moves in accordance with the movement of the user terminal 10b. Furthermore, the virtual route 404a changes to a virtual route 404b corresponding to the position of the waypoint 408b after the movement. The change in the virtual route 404a causes the route along which the moving object 20 actually moves to be adjusted. In this way, in this embodiment, the route of the moving object 20 is adjusted based on the operation of the user U2 to move the user terminal 10a. In this way, the route of the moving object 20 is adjusted, and new movement information representing the adjusted route is generated. Based on the movement information, the moving object 20 becomes able to fly around the tower 420.

[0056] Here, the case where a waypoint is set in advance on the virtual route 404a has been described. If a waypoint is not set in advance on the virtual route 404a, the user U2 can also set a waypoint by, for example, touching a part of the image 614 of the virtual route 404 displayed on the display screen 610. Waypoints set in this way can also be adjusted by the method described above.

[0057] As described above, according to this embodiment, the user U2 can fine-tune the waypoint by operating the display screen 610 and moving the user terminal 10a after the waypoint has been set. This makes it possible to generate information for moving the moving object 20 more intuitively.

[0058] The above has described a method for generating new travel information by adjusting the virtual route 404a of the moving object 20 when the virtual route 404a has been set in advance. Next, two methods for setting the route of the moving object 20 when the route of the moving object 20 has not been set in advance will be described with reference to Figures 13 and 14. Figures 13 and 14 are diagrams showing how a new route for the moving object 20 is set based on the operation of the user U2.

[0059] In either method, the user U2 operates the user terminal 10 to set the route and waypoints of the moving object 20. The waypoints set by the method described with reference to Figures 13 and 14 may also be adjusted by the method described above with reference to Figures 8 to 12.

[0060] A first method for setting the route of the moving object 20 will be described with reference to FIG. 13. First, the user U2 touches a part of the display screen 610c of the user terminal 10c (for example, a designated point 616c displayed on the display screen 610c). While touching the designated point 616c, the user U2 moves the user terminal 10c, for example, downward as indicated by the dashed line. The user terminal 10c stores the trajectory of the movement of the user terminal 10c as the route of the moving object 20. At this time, the user terminal 10c may set waypoints on the route and store them together with the route.

[0061] Next, a second method for setting a route for the moving object 20 will be described with reference to Fig. 14. In the second method, the user U2 sets a Waypoint 408d by touching a display screen 610d of the user terminal 10d, as shown in the upper part of Fig. 14. An image 616d of the set Waypoint 408d is displayed on the display screen 610d.

[0062] 14, the user U2 moves the position of the user terminal 10d and touches the display screen 610e of the user terminal 10e at the new position to set a waypoint 408e. Thereafter, the movement of the user terminal 10 and the setting of waypoints 408 are repeated, and the set multiple waypoints 408 are connected to generate a route for the moving object 20. The generated route and waypoints 408 are stored in the user terminal 10.

[0063] In this way, according to this embodiment, even if the route of the moving body 20 has not been set in advance, the user U2 can set the route of the moving body 20 by operating the display screen 610 and moving the user terminal 10.

[0064] Here, the position of a Waypoint that is set by operating the user terminal 10 will be described in more detail with reference to FIGS. 15 to 18. FIG. 15 is a diagram showing how the position of the imaging unit 110a included in the user terminal 10f is set as the Waypoint 410. FIG. 16 is a diagram showing a display screen 610f when the position of the imaging unit 110a included in the user terminal 10f is set as the Waypoint 410. FIG. 17 is a diagram showing how a position that is a predetermined distance away from the user terminal 10g is set as the Waypoint 412. FIG. 18 is a diagram showing the display screen 610 when a position that is a predetermined distance away from the user terminal 10g is set as the Waypoint 412.

[0065] First, the position of the Waypoint to be set will be described with reference to Fig. 15 and Fig. 16. As shown in Fig. 15, a virtual tower object 422a is placed on a desk 500. The imaging unit 110a included in the user terminal 10f captures an image of the area in front of the imaging unit 110a. In other words, the imaging unit 110a captures an image of the range that includes the virtual object 422a. Here, the position of the imaging unit 110a is specified as the Waypoint.

[0066] At this time, as shown in Fig. 16, an image 612f of a virtual object of a tower placed on the desk 500 is displayed on a display screen 610f of the user terminal 10f. The user can set a Waypoint by, for example, touching the display screen 610f while looking at the display screen 610f. The user can also move the user terminal 10f and set a Waypoint 410 at a new position.

[0067] At this time, the image displayed on the display screen 610 may correspond to an image actually captured at the waypoint by the imaging device of the moving object 20. In this case, the user can check in advance the image captured by the imaging device provided in the moving object 20.

[0068] 17 and 18, a method for the user terminal 10g to set a position a predetermined distance away from the image capturing unit 110a as a Waypoint will be described. Specifically, a position a distance d forward from the image capturing unit 110a and slightly shifted downward from the optical axis 414 of the image capturing unit 110a is set as Waypoint 412.

[0069] 18, an image 612g of a virtual tower object and an image 616 of a waypoint are displayed on a display screen 610g of the user terminal 10g. Furthermore, a guide surface 617 that connects the user terminal 10g and the image 616 of the waypoint is displayed on the display screen 610g. While viewing the image 616 of the waypoint arranged on the guide surface 617, the user can set the waypoint 412 by performing a touch operation on the display screen 610g, for example.

[0070] At this time, since the waypoint 412 is located below the optical axis 414 of the imaging unit 110a, it is believed that the user can more easily recognize the position of the waypoint 412 by referring to the display screen 610g.

[0071] The above describes a method for setting a waypoint by operating the display screen 610. Next, variations on the method for setting a waypoint will be described with reference to Figures 19 and 20. Specifically, a method for setting a waypoint using a designated object that specifies the route of the moving object 20 will be described.

[0072] Fig. 19 is a diagram showing how a waypoint is set using the designation stick 620. Fig. 20 is a diagram showing a display screen 610h when a waypoint is set using the designation stick 620.

[0073] In this embodiment, a spherical designation object 622 is provided at the tip of the designation wand 620. Here, the designation wand 620 may be a touch pen or the like that can touch the user terminal 10 to perform various operations. Furthermore, the user terminal 10h is assumed to be equipped with a sensor that can detect the three-dimensional position of the designation object 622 provided on the designation wand 620. Specifically, the user terminal 10h is equipped with a distance measurement sensor such as a ToF sensor or a stereo camera. The user terminal 10 acquires position information of the designation object 622 based on sensor information from the distance measurement sensor. The position information of the designation object 622 is expressed in three dimensions (x, y, z). Here, z is the direction of gravity (up and down). The x and y directions are perpendicular to the z direction and are orthogonal to each other.

[0074] Here, a Waypoint is set based on the position information. Specifically, for example, when the user performs a touch operation on the display screen 610h, the user terminal 10 sets the position of the specified object 622 as a Waypoint.

[0075] At this time, the image 618 of the designation stick and the image 616 of the designated object are displayed on the display screen 610h of the user terminal 10h. Therefore, the user can set the Waypoint while checking the position of the designated object 622 on the display screen 610h.

[0076] <<3.2. Generation of imaging information>> The above describes the operation for generating movement information (more specifically, information including Waypoints) for controlling the movement of the moving body 20. Next, two methods for generating imaging information for controlling the range captured by the imaging device provided in the moving body 20 will be described with reference to Figs. 21 and 22. Fig. 21 is a diagram showing how the orientation of the imaging device of the moving body 20 is set by moving the orientation of the user terminal 10i. Fig. 22 is a diagram showing how the angle of view of the imaging device of the moving body 20 is set by performing a pinch operation on the display screen of the user terminal 10j.

[0077] First, with reference to FIG. 21, a method for setting the direction in which the imaging device of the moving object 20 captures images will be described. For example, the user selects one of the waypoints included in the route of the moving object 20. In this state, as shown in FIG. 21, the user can adjust the direction in which the imaging unit 110a included in the user terminal 10i captures images (i.e., the imaging ranges 134a, 134b) by moving the orientation of the user terminal 10i. Direction information regarding the adjusted direction in which the imaging unit 110a captures images is generated as imaging information. In other words, the user terminal 10i can generate direction information based on attitude information of the user terminal 10i. Based on the direction information, the imaging device of the moving object 20 can capture images in the same direction as the adjusted imaging unit 110a at the set waypoint.

[0078] Furthermore, the user terminal 10 according to this embodiment can generate angle-of-view information for controlling the angle of view of the imaging device of the moving object 20 based on a pinch-out operation or a pinch-in operation on the display screen by the user.

[0079] Next, a method for setting the angle of view of the imaging device of the moving object 20 will be described with reference to Fig. 22. The user selects, for example, one of the waypoints included in the route of the moving object 20. In this state, the user performs a pinch-out or pinch-in operation on the display screen of the user terminal 10j to adjust the imaging range 134 of the imaging unit 110a, thereby setting the angle of view of the imaging device when the moving object 20 passes through the selected waypoint. In other words, the user terminal 10j can generate angle-of-view information for controlling the angle of view of the imaging device of the moving object 20 based on the user's pinch-out or pinch-in operation on the display screen.

[0080] The above has described the generation of direction information and angle of view information by the user terminal 10 based on user operations. The imaging device of the moving body 20 can capture images based on the direction information and angle of view information. Note that the direction information and angle of view information may be generated when the position of the waypoint is set, or may be generated after the position of the waypoint is set.

[0081] <<3.3. Simulation of moving object movements>> Next, a simulation of the operation of the moving object 20 by the user terminal 10 based on movement information and imaging information will be described with reference to Fig. 23 and Fig. 24. Specifically, the user terminal 10 simulates the movement of the moving object 20 and images captured by an imaging device provided in the moving object 20. Fig. 23 is a diagram showing a display screen 611 that displays the simulation results of the movement of the moving object 20. Fig. 24 is a diagram showing a display screen 610k that displays the simulation results of images captured by an imaging device provided in the moving object 20.

[0082] 23, an image 612i of a virtual object of a tower placed on a desk and an image 630 of a moving object represented schematically by a triangle are displayed on a display screen 611 of the user terminal 10. When a simulation of the movement of the image 630 of the moving object is started, the image 630 of the moving object moves along a virtual route 615 connected by waypoint images 616a-m. By checking the movement of the image 630 of the moving object, the user can predict how the moving object 20 will actually move.

[0083] Furthermore, the user terminal 10k according to this embodiment can also simulate an image captured by an imaging device included in the moving object 20. Specifically, the user terminal 10k can display an image predicted to be captured by the imaging device of the moving object 20 when the moving object 20 flies through the waypoint set as described above. As shown in FIG. 24, an image predicted to be captured is displayed on a display screen 610k of the user terminal 10k. By looking at the display screen 610k, the user can predict the image captured by the imaging device included in the moving object 20. Note that the user can also stop the video displayed on the display screen 610k by, for example, touching a stop button 619 shown in the center of the display screen 610k.

[0084] <4. Imaging method> Below, we will explain a method for capturing scenery images using the moving body 20. First, we will explain three methods for capturing scenery images without using the technology of the present disclosure. Then, we will explain a method for capturing scenery images using the moving body 20 using the technology of the present disclosure.

[0085] In the following description, it is assumed that the mobile object 20 (e.g., a drone) is flown around a structure such as a tower to capture impressive footage for, for example, a commercial. In such a case, it is necessary to fly the mobile object 20 in a three-dimensional space. Therefore, impressive footage can only be captured by appropriately controlling various conditions of the mobile object 20, such as the flight position, speed, and camera direction. Therefore, advanced techniques for operating the mobile object 20 are required to capture impressive footage.

[0086] Furthermore, it is difficult for a user to manually fly the moving object 20 along the same trajectory multiple times. Furthermore, when capturing images outdoors or capturing images of a wide range, it is necessary to consider the lighting conditions and the comings and goings of people, making the timing of capturing images important. For this reason, by capturing images repeatedly, images can be captured under good conditions.

[0087] <<4.1. Manual Imaging Method>> First, with reference to Fig. 25, a method in which a user manually controls the moving body 20 using a control device and causes the imaging device of the moving body 20 to capture a landscape will be described. Here, it is assumed that the control device controls the movement of the moving body 20 and the orientation of the imaging device provided in the moving body 20. Fig. 25 is a flowchart showing the imaging method by manual operation. Below, the imaging method by manual operation will be described with reference to the flowchart shown in Fig. 25.

[0088] First, the user checks the difference in the image depending on the imaging conditions (step S101). More specifically, the user actually flies the moving body 20 around the building by manual operation, and checks the difference in the appearance of the image depending on the imaging conditions, such as the orientation of the imaging device of the moving body 20 or the distance between the moving body 20 and the building. It is preferable that the user is familiar with operating the moving body 20.

[0089] Next, the user captures video using the moving body 20 (step S103). More specifically, the user manually controls the flight of the moving body 20 and the orientation of the imaging device so that impressive video is captured, and causes the imaging device of the moving body 20 to capture video.

[0090] At this time, the user may display a two-dimensional map screen on various known mobile terminals such as a tablet terminal together with the image captured by the imaging device provided in the mobile object 20, and display the route along which the mobile object 20 is flying. Furthermore, the user may set waypoints on the route based on predetermined rules. This allows the user to set waypoints while checking the captured image.

[0091] Next, the user checks the video captured in step S103 (step S105). If the flight of the moving body 20 and the orientation of the imaging device have been controlled as intended (step S107: YES), the process proceeds to step S109. On the other hand, if the flight of the moving body 20 and the orientation of the imaging device have not been controlled as intended (step S107: NO), the process returns to step S103.

[0092] Even if the flight of the moving body 20 and the orientation of the imaging device are controlled as intended (step S107: YES), if the intended impressive image is not captured because, for example, the sun goes down or an unintended person crosses in front of the imaging device (step S109: NO), the process returns to step S103. On the other hand, if the intended impressive image is captured (step S109: YES), the imaging method shown in Fig. 25 ends.

[0093] The above describes a method for capturing images by manual operation. With this method, in order to obtain the intended image, it is necessary to repeatedly repeat the flight of the moving body 20 and the orientation of the image capture device by manual operation. Therefore, in order to obtain the desired image, it is time-consuming and labor-intensive, and each time an image is captured, manpower is required.

[0094] <<4.2. Imaging Method Using Automatic Flight>> Next, a method for making the imaging device capture images while making the moving body 20 fly automatically will be described with reference to Fig. 26. Fig. 26 is a flowchart showing a method for making the imaging device capture images while making the moving body 20 fly automatically. The following description will be given with reference to the flowchart shown in Fig. 26.

[0095] First, the processes of steps S201 to S207 are carried out. However, since the processes of steps S201 to S207 are substantially the same as the processes of steps S101 to S107, the description thereof will be omitted here.

[0096] If the flight of the moving body 20 and the orientation of the imaging device are controlled as intended (step S207: YES), data on the imaging conditions is saved (step S209). More specifically, when the flight of the moving body 20 and the orientation of the imaging device are controlled as intended, various imaging conditions such as the position, speed, and orientation of the imaging device when the moving body 20 is flying are recorded. The imaging conditions are recorded on various known recording media provided in the moving body 20, etc. Note that information regarding the position, speed, etc. of the moving body 20 is acquired by a GPS, IMU, etc. provided in the moving body 20.

[0097] Next, the imaging operation is reproduced (step S211). More specifically, the flight of the moving body 20 and the orientation of the imaging device are automatically reproduced based on the imaging conditions recorded in step S209. The image captured at this time is confirmed by the user.

[0098] If the intended image is not captured (step S213: NO), the process returns to step S211 and the imaging operation is repeated. On the other hand, if the sunlight conditions are right and the intended image is captured (step S213: YES), the imaging shown in FIG. 26 ends.

[0099] The imaging method by automatic flight has been described above. This method reduces the burden on the user because the user does not need to manually operate the same moving object 20.

[0100] It is also possible to display a map on a display screen of a tablet terminal or the like based on the data recorded in step S209, depict a virtual route of the moving object 20 on the map, and fine-tune the virtual route. However, when the virtual route is displayed on a two-dimensional map screen, it is difficult to intuitively adjust, for example, the altitude of the virtual route.

[0101] Furthermore, since the self-position of the moving body 20 is calculated using a GPS (Global Positioning System) or an IMU, an error of about 50 cm to 1 m occurs in the relative position to a building or the like, which depends on the GPS.

[0102] <<4.3. Imaging method using a map displayed on the device screen>> In the above method, the user needs to actually go to the location where the moving body 20 will fly, and operate the moving body 20 and set waypoints. Therefore, a method can be considered in which the user specifies a route for the moving body 20 to fly in advance using a map displayed on a display screen of a tablet terminal or the like, and then causes the moving body 20 to fly along the specified route.

[0103] More specifically, the user sets a waypoint by, for example, displaying a map on the display screen of a tablet device and touching the display screen. The position of the waypoint may be set using longitude and latitude. At this time, the user may also set the speed and altitude of the moving body 20 at the specified waypoint. Furthermore, the user may also set the orientation of the moving body 20 at the specified waypoint. For example, the user may set the orientation of the moving body 20 to face the direction of travel.

[0104] Similarly, multiple waypoints are set and these waypoints are connected to set a route for the moving body 20. By recording or transmitting information representing the set route to the moving body 20, the user can make the moving body 20 fly along the set route and cause the imaging device to capture images.

[0105] According to this method, the user can set the route along which the moving body 20 will fly before going to the site where the moving body 20 will fly. Therefore, the user can set the route along which the moving body 20 will fly while staying at home, at work, or at home. However, the type of image captured by the imaging device of the moving body 20 cannot be known until the moving body 20 is actually flown and the imaging device is caused to capture the image. Furthermore, it cannot be known how the image captured by the imaging device will change when the position of the waypoint is corrected until the moving body 20 is actually flown and the imaging device is caused to capture the image.

[0106] Furthermore, setting a Waypoint by touching the map displayed on the display screen is convenient when flying the mobile object 20 over a wide area. However, when the imaging device of the mobile object 20 is to capture dynamic images of the surroundings of a building, it is considered difficult to set the detailed route of the mobile object 20. Furthermore, because a two-dimensional map is displayed on the display screen, the altitude of the mobile object 20 must be set numerically, making it difficult to intuitively set a Waypoint.

[0107] <<4.4. Imaging method according to the present disclosure>> Next, an imaging method according to the present disclosure will be described with reference to Figs. 27 to 29. Fig. 27 is a flowchart showing the procedure up to generation of a virtual object. Fig. 28 is a flowchart showing the procedure up to imaging of a video based on the generated movement information and imaging information. Fig. 29 is a diagram showing display processing by the information processing device 100. Hereinafter, an imaging method according to the present disclosure will be described with reference to Figs. 27 to 29. In the following explanation, the above-mentioned Figs. 2 to 24 will be appropriately referred to.

[0108] 27 is substantially the same as the user's operation in step S101. However, in the imaging method according to the present disclosure, the user operating the moving body 20 may be unfamiliar with operating the moving body 20.

[0109] Next, the user causes the imaging device of the moving body 20 to capture an image (step S303). The user causes the imaging device of the moving body 20 to capture an image of a subject that will be the basis of the virtual object. At this time, the user may manually control the flight of the moving body 20 and the orientation of the imaging device so that an impressive image is captured, and cause the imaging device of the moving body 20 to capture the image. For example, the user may cause the moving body 20 to circle around a tower 420 as shown in FIG. 4 and cause the imaging device 206 to capture the image. At this time, the imaging device 206 is assumed to capture an image including the tower 420 and the forest 430.

[0110] Next, the user checks the captured image (step S305). More specifically, the user checks that the image captured by the image capturing device 206 is the intended image.

[0111] Next, a virtual object is generated (step S307). More specifically, information about a three-dimensional virtual object is generated using various known CG techniques based on information such as the image captured in step S303, the position and posture of moving body 20 when the image was captured, and the orientation of imaging device 206. For example, information about virtual tower object 422 shown in FIG. 5 and virtual forest object 432 shown in FIG. 6 is generated.

[0112] The position and orientation of the moving body 20 may be calculated more accurately not only using the GPS and IMU but also using bundle adjustment in the process of generating a three-dimensional virtual object based on captured images. This allows the relative position or orientation of the moving body 20 with respect to the environment, such as buildings, to be calculated more accurately. Here, bundle adjustment is a method of estimating various parameters with high accuracy from images. Information about the generated virtual object is recorded in the storage unit 180 included in the user terminal 10. At this time, information about the route 402 and waypoints 406 traveled by the moving body 20 as shown in FIG. 7 may also be recorded in the storage unit 180.

[0113] The processing up to the generation of a virtual object has been described above with reference to Fig. 27. Next, the procedure up to the capture of a desired video will be described with reference to Fig. 28 and Fig. 29. Note that the processing in steps S401 to S405 shown in Fig. 28 and Fig. 29 is processing mainly performed by the information processing device 100 according to an embodiment of the present disclosure.

[0114] The information processing device 100 performs a process of displaying a virtual object (step S401). The process of displaying a virtual object will be described with reference to FIG. 29. FIG. 29 is a flowchart showing the process of displaying a virtual object. The process of displaying a virtual object will be described below with reference to the flowchart shown in FIG. 29. The process shown in FIG. 29 is executed, for example, when the start button 602 displayed on the display screen 610 of the user terminal 10a is touched, as described with reference to FIG. 8.

[0115] First, the acquisition unit 140 acquires image information and sensor information (step S501). More specifically, the acquisition unit 140 acquires image information including the desk 500 captured by the imaging unit 110. The acquisition unit 140 also acquires IMU information detected by the sensor unit 120 or distance information from the user terminal 10a to the desk 500. The acquisition unit 140 transmits the acquired image information and distance information to the detection unit 151 included in the processing unit 150. The acquisition unit 140 also transmits the acquired image information, distance information, and IMU information to the self-position calculation unit 154 included in the processing unit 150.

[0116] Next, the plane detection unit 152 detects a plane based on the image information and distance information transmitted from the acquisition unit 140 (step S503). Here, the plane detection unit 152 detects a flat plane 506 on the desk 500. The plane detection unit 152 transmits the detection result to the virtual object calculation unit 155.

[0117] Next, the self-position calculation unit 154 calculates the self-position of the user terminal 10 based on the image information, distance information, and IMU information (step S505). More specifically, the self-position calculation unit 154 calculates the position and orientation of the user terminal 10 with respect to the desk 500 or the environment around it. The self-position calculation unit 154 transmits the calculation result to the virtual object calculation unit 155.

[0118] Next, the virtual object calculation unit 155 calculates the position, direction, scale, etc. of the virtual object to be placed based on the calculation result of the self-position calculation unit 154 and the information related to the virtual object recorded in the storage unit 180 (step S507). The virtual object calculation unit 155 transmits the calculation result to the movement information generation unit 157.

[0119] Next, the movement information generation unit 157 sets a route for the moving object 20 based on the calculation result of the virtual object calculation unit 155 and the Waypoint information recorded in the storage unit 180 (step S509). For example, the movement information generation unit 157 sets a virtual route that circles around a virtual object placed on the desk 500. The movement information generation unit 157 transmits information about the set virtual route to the display information generation unit 159.

[0120] Next, the display information generation unit 159 generates display information (step S511). More specifically, the display information generation unit 159 generates display information that displays a virtual path of the moving object 20 around the virtual object placed on the desk 500, and transmits the generated display information to the display control unit 170.

[0121] Next, the display control unit 170 controls the display of the display unit 175 so that an image of the virtual route is displayed around the virtual object placed on the desk 500 (step S513). As a result, an image 612 of the virtual tower object 422 and an image 614 of the virtual route circling around the virtual object 422 are displayed on the display screen of the display unit 175 on the desk 500 in front of the user.

[0122] The display processing of the virtual object has been described above with reference to Fig. 29. Next, returning to Fig. 28, the imaging method according to the present disclosure will be described.

[0123] The information processing device 100 generates movement information and imaging information (step S403). For example, as described with reference to FIGS. 9 to 22, movement information such as waypoints and imaging information such as the orientation and zoom rate of the imaging device are generated based on an operation by the user to move the user terminal 10, and are transmitted to the prediction unit 160. Here, the processing of the information processing device 100 in the operations described with reference to FIGS. 9 to 22 will be described.

[0124] (Process to adjust pre-set waypoints) Here, a description will be given of the processing of the information processing device 100 in the processing for adjusting a Waypoint described with reference to Figures 9 to 12. As shown in Figure 9, when the user U2 touches the image 616a of the Waypoint displayed on the display screen 610a, the input unit 130 transmits, to the movement information generation unit 157, input information indicating that the Waypoint 408a corresponding to the image 616a of the Waypoint has been selected.

[0125] 11 and 12, when the user U2 pulls the user terminal 10 toward the user U2, the sensor unit 120 detects the movement of the user terminal 10 and transmits the detected sensor information to the self-position calculation unit 154. The self-position calculation unit 154 calculates the position and attitude of the user terminal 10 based on the sensor information. The self-position calculation unit 154 transmits the calculation result to the movement information generation unit 157.

[0126] Next, the movement information generation unit 157 corrects the virtual route of the moving object 20 so that the position of the selected waypoint 408a is displaced by the distance moved by the user terminal 10. As a result, information about the new virtual route is generated as movement information and transmitted to the prediction unit 160.

[0127] (Process to set a new Waypoint) Next, a description will be given of the processing performed by the information processing device 100 in the operations described with reference to Fig. 13 or 14. The acquisition unit 140 acquires input information based on an operation by a user on the display screen 610 from the input unit 130. The acquisition unit 140 also acquires image information from the imaging unit 110 and distance information and IMU information from the sensor unit 120. The acquisition unit 140 transmits the acquired information to the processing unit 150.

[0128] The self-position calculation unit 154 calculates the self-position of the user terminal 10 based on the transmitted sensor information or distance information, etc., and transmits the calculation result to the generation unit 156. The movement information generation unit 157 identifies the position of a waypoint based on the calculation result and input information, etc. The movement information generation unit 157 sets a virtual route for the moving object 20 by connecting the identified multiple waypoints, and transmits the virtual route to the prediction unit 160 as movement information.

[0129] (Process of setting a Waypoint using the designation stick) Next, the processing of the information processing device 100 when setting a Waypoint using the designation wand 620, which has been described with reference to Figures 19 and 20, will be described. First, the acquisition unit 140 acquires IMU information and sensor information from the sensor unit 120. Furthermore, the acquisition unit 140 acquires image information from the imaging unit 110 and transmits the image information to the object detection unit 153.

[0130] The object detection unit 153 detects the designated object 622 included in the image based on the image information. Furthermore, the object detection unit 153 detects, for example, the distance and direction from the imaging unit 110 to the designated object 622 based on the sensor information, and transmits the detection result to the generation unit 156.

[0131] The movement information generation unit 157 identifies the position of the specified object 622 based on the result of detection by the object detection unit 153, and sets the position as a Waypoint. The movement information generation unit 157 sets a virtual route by connecting the set multiple Waypoints, and transmits the virtual route to the prediction unit 160 as movement information.

[0132] (Process for setting the direction of the imaging device) Next, the processing of the information processing device 100 when setting the direction of the imaging device, as described with reference to Fig. 21, will be described. The self-position calculation unit 154 calculates the attitude of the user terminal 10 based on sensor information and transmits the calculation result to the generation unit 156. The imaging information generation unit 158 ​​sets the direction of the imaging device based on the calculated attitude. The imaging information generation unit 158 ​​transmits the set direction of the imaging device to the prediction unit 160 as direction information.

[0133] (Process for setting the angle of view of the imaging device) Next, the process of the information processing device 100 when setting the angle of view of the imaging device, which has been described with reference to FIG. 22, will be described.

[0134] The imaging information generation unit 158 ​​acquires input information indicating that a pinch-in operation or a pinch-out operation has been performed by the user from the input unit 130. The imaging information generation unit 158 ​​generates field of view information indicating the field of view of the imaging device based on the input information, and transmits the field of view information to the prediction unit 160.

[0135] Next, the information processing device 100 performs a simulation of the movement of the moving object 20 and the image captured by the imaging device (step S405). For example, the prediction unit 160 simulates the movement of the moving object 20 on the display screen 611. Specifically, the movement prediction unit 161 predicts the movement of the moving object 20 based on the movement information, and transmits the prediction result to the display information generation unit 159.

[0136] Furthermore, the prediction unit 160 simulates the captured image. More specifically, the capture prediction unit 162 predicts the image captured by the moving object 20 based on the movement information and the capture information, and transmits the prediction result to the display information generation unit 159.

[0137] Next, the display unit 175 displays the prediction result (step S407). More specifically, the display information generation unit 159 generates display information for displaying the prediction result based on the prediction result, and transmits the display information to the display control unit 170. The display control unit 170 controls the display of the display unit 175 based on the display information so that the display unit 175 displays the prediction result. As a result, the prediction result is displayed on the display unit 175. More specifically, the prediction result of the movement of the moving object 20 shown in FIG. 23 or the prediction result of the video captured by the imaging device of the moving object 20 shown in FIG. 24 is displayed.

[0138] Next, if the simulation was performed as intended (step S409: YES), the process proceeds to step S411. At this time, the storage unit 180 may store the movement information and imaging information used in the simulation. On the other hand, if the simulation was not performed as intended (step S409: NO), the process returns to step S403.

[0139] Next, the moving body 20 moves and an image is captured by the imaging device (step S411). For example, the virtual route of the moving body 20 formed in step S403 is converted into a real route along which the moving body 20 actually moves by converting it into a coordinate system of real space by the movement information generation unit 157. Information about the real route is transmitted to the moving body 20 by the communication control unit 190. While the moving body 20 flies along the generated real route, the imaging device captures images of the scenery based on the imaging information.

[0140] If an impressive image is captured as intended (step S413: YES), the imaging process shown in Fig. 28 ends. On the other hand, if an impressive image is not captured as intended (step S413: NO), the process returns to step S411.

[0141] <5. Effects> The imaging method according to the present disclosure has been described above. The information processing device 100 according to the present disclosure controls the display of a virtual object based on an object existing in real space on a display screen, and generates movement information for controlling the movement of a moving object. Therefore, when a user wants to fly a moving object 20, such as a drone, around an object existing in real space that is the basis of the virtual object, the user can specify a path for the moving object 20 while viewing the virtual object. Therefore, the information processing device 100 according to this embodiment makes it possible to generate movement information for controlling the movement of the moving object 20 more intuitively.

[0142] Furthermore, according to the information processing device 100 of this embodiment, the movement information generation unit 157 generates movement information based on an operation by a user U2 viewing the display screen 610. The user U2 can specify movement information such as the route of the moving object 20 while viewing the virtual object displayed on the display screen 610. This makes it possible to generate movement information for controlling the movement of the moving object 20 more intuitively.

[0143] 9 and 11, the display on the display screen 610 includes an image 614 of the virtual route of the moving object 20. This makes it easier for the user U2 to imagine the route of the moving object 20.

[0144] Furthermore, in this embodiment, at least a portion of the image 614 of the virtual route displayed on the display screen 610 displays an image 616 of one or more waypoints (adjustment portions) for adjusting the route of the moving object 20. The movement information generation unit 157 generates movement information based on an operation of moving the image 616 of the waypoint. Therefore, the user U2 can specify the route of the moving object 20 simply by moving the image 616 of the waypoint that serves as a landmark on the route, thereby making it possible to generate movement information for controlling the movement of the moving object 20 more intuitively.

[0145] 13, in this embodiment, the movement information generation unit 157 moves the position of the Waypoint 408 based on an operation to move the position of the display screen 610. This allows the user to specify the route of the moving object 20 more intuitively.

[0146] Furthermore, in this embodiment, the image 612 of the virtual object is displayed superimposed on an image captured by the imaging unit 110 included in the user terminal 10. This allows the user U2 to recognize the virtual object 422a as if it were present in real space. This allows the user U2 to more intuitively specify the route of the moving object 20. Furthermore, by aligning the viewpoint of the imaging unit 110 with the waypoint, it is also possible to display an image captured from the waypoint on the display screen. This makes it possible to predict in advance how the captured image will change when the position of the waypoint changes.

[0147] Also, in this embodiment, as described with reference to FIGS. 15 to 18 , the movement information generation unit 157 generates movement information based on an operation by the user U2 to move the viewpoint of the imaging unit 110. More specifically, the movement information generation unit 157 generates movement information based on a movement of a predetermined position based on the viewpoint. The display screen 610 includes an image captured by the imaging unit 110. This makes it easier for the user U2 to imagine the scenery captured by the imaging device included in the moving object 20 when the moving object 20 actually moves, and makes it possible for the imaging device included in the moving object 20 to capture more desired images.

[0148] 15 and 16, the route of the moving object 20 may be the viewpoint of the image capturing unit 110. Alternatively, the route of the moving object 20 may be a position a predetermined distance away from the viewpoint of the image capturing unit 110. For example, as described with reference to FIG. 17, a position a distance d forward from the viewpoint of the image capturing unit 110 and lowered from the optical axis of the image capturing unit 110 to an extent that it fits within the angle of view may be specified as the route of the moving object 20. In this case, by displaying waypoints and the like on the display screen 610g as shown in FIG. 18, the user U2 can more intuitively specify the route of the moving object 20.

[0149] Furthermore, in this embodiment, as described with reference to FIGS. 19 and 20 , the movement information generation unit 157 generates movement information based on an operation of moving a designated object that designates the route of the moving object 20. More specifically, in this embodiment, the route of the moving object 20 is designated based on the route of the designated object 622 attached to the tip of the designation wand 620. This allows the user to designate the route of the moving object 20 through the simple operation of moving the designated object 622. Furthermore, as shown in FIG. 20 , an image 616 of the designated object 622 is displayed on the display screen 610h. This allows the user U2 to recognize the position of the designated object 622 via the display screen 610h. This allows the user U2 to more easily imagine the route of the moving object 20.

[0150] In the present embodiment, the moving object 20 is equipped with an imaging device. The imaging device captures images of the scenery around the moving object 20. The information processing device 100 according to the present embodiment also includes an imaging information generation unit 158 ​​that generates imaging information for controlling the range captured by the imaging device included in the moving object 20, based on a user operation. This allows the user to specify the range captured by the imaging device included in the moving object 20, based on various operations, thereby allowing the imaging device of the moving object 20 to capture more appropriate images.

[0151] Furthermore, in this embodiment, the imaging information generation unit 158 ​​generates, as imaging information, direction information regarding the direction in which imaging is performed by the imaging device provided in the moving object 20. This allows the user to make the imaging device of the moving object 20 capture more appropriate images.

[0152] In this embodiment, an image captured by the imaging unit 110 is displayed on the display screen. As described with reference to Fig. 21 , the imaging information generation unit 158 ​​generates direction information based on an operation to move the orientation of the imaging unit 110. This allows the user to generate direction information while guessing the image captured by the imaging device of the moving object 20, thereby allowing the imaging device to capture more appropriate images.

[0153] Furthermore, in this embodiment, the imaging information generation unit 158 ​​can generate, as imaging information, field of view information for controlling the field of view of the imaging device of the moving object 20, based on a pinch-out operation or a pinch-in operation on the display screen by the user U2. This allows the user to easily specify the range to be captured by the imaging device of the moving object 20.

[0154] Moreover, in this embodiment, the information processing device 100 further includes a movement prediction unit 161 that predicts the movement of the moving object 20 based on the movement information. More specifically, the movement prediction unit 161 can simulate the movement of the moving object 20. Therefore, the user can check the route of the moving object 20 in advance based on the simulation of the movement of the moving object 20. In this embodiment, as described with reference to FIG. 23 , the simulation result of the movement of the moving object 20 (i.e., the prediction result) is displayed on the display screen 611. Therefore, the user can more easily check the route of the moving object 20 by looking at the display screen 611.

[0155] Moreover, in the present embodiment, the information processing device 100 further includes an imaging prediction unit 162 that predicts an image to be captured by an imaging device of the moving object 20, based on movement information and imaging information. In the present embodiment, the imaging prediction unit 162 can simulate an image to be captured by the imaging device. The user can check the image to be captured based on the result of the simulation. Furthermore, in the present embodiment, as described with reference to FIG. 24 , the result of the simulation by the imaging prediction unit 162 is displayed on the display screen 610k. Therefore, the user can easily check the prediction result by the imaging prediction unit 162.

[0156] Furthermore, in this embodiment, the moving object 20 is capable of moving three-dimensionally. Therefore, the user can three-dimensionally specify the route of the moving object 20. This allows the user to generate movement information for controlling the movement of the moving object 20 more intuitively.

[0157] Furthermore, according to this embodiment, once the route of the moving body 20 is set, it is possible to fly the moving body 20 along the same route multiple times without human intervention, and to have the imaging device capture the same image multiple times.

[0158] <6. Hardware Configuration> Next, an example of a hardware configuration of a user terminal 10 constituting the information processing system 1 according to an embodiment of the present disclosure, such as the above-described user terminal 10, will be described in detail with reference to Fig. 30. Fig. 30 is a functional block diagram showing an example of a hardware configuration of the user terminal 10 constituting the information processing system 1 according to an embodiment of the present disclosure.

[0159] The user terminal 10 constituting the information processing system 1 according to this embodiment mainly includes a CPU 901, a ROM 902, and a RAM 903. The user terminal 10 further includes a host bus 904, a bridge 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device 910, a drive 912, a connection port 914, and a communication device 916.

[0160] The CPU 901 functions as an arithmetic processing device and control device, and controls all or part of the operations within the user terminal 10 in accordance with various programs recorded in the ROM 902, RAM 903, storage device 910, or removable recording medium 913. The ROM 902 stores programs used by the CPU 901, calculation parameters, etc. The RAM 903 temporarily stores programs used by the CPU 901, parameters that change as appropriate during program execution, etc. These are interconnected by a host bus 904 constituted by an internal bus such as a CPU bus. For example, the acquisition unit 140, processing unit 150 (each functional unit shown in FIG. 3), display control unit 170, and communication control unit 190 shown in FIG. 2 can be constituted by the CPU 901.

[0161] The host bus 904 is connected to an external bus 906, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 905. In addition, an input device 908, an output device 909, a storage device 910, a drive 912, a connection port 914, and a communication device 916 are connected to the external bus 906 via an interface 907.

[0162] The input device 908 is an operation means operated by the user, such as a mouse, keyboard, touch panel, button, switch, lever, pedal, etc. The input device 908 may be, for example, a remote control means (so-called remote control) that uses infrared or other radio waves, or an externally connected device 915 such as a mobile phone or PDA that is compatible with the operation of the user terminal 10. The input device 908 is further composed of, for example, an input control circuit that generates an input signal based on information input by the user using the above operation means and outputs the signal to the CPU 901. The user of the user terminal 10 can input various data to the user terminal 10 and instruct processing operations by operating the input device 908.

[0163] The output device 909 is configured with a device capable of visually or audibly notifying the user of acquired information. Such devices include display devices such as CRT display devices, liquid crystal display devices, plasma display devices, EL display devices, and lamps, audio output devices such as speakers and headphones, and printer devices. The output device 909 outputs, for example, the results obtained by various processes performed by the user terminal 10. Specifically, the display device displays the results obtained by various processes performed by the user terminal 10 as text or images. On the other hand, the audio output device converts audio signals consisting of reproduced voice data, acoustic data, etc. into analog signals and outputs them.

[0164] The storage device 910 is a data storage device configured as an example of a storage unit of the user terminal 10. The storage device 910 is configured, for example, by a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 910 stores programs executed by the CPU 901, various data, and the like. For example, the storage unit 180 shown in FIG. 2 can be configured by the storage device 910.

[0165] The drive 912 is a reader / writer for a recording medium, and is built into or externally attached to the user terminal 10. The drive 912 reads information recorded on a removable recording medium 913, such as an attached magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 903. The drive 912 can also write information to the attached removable recording medium 913, such as an attached magnetic disk, optical disk, magneto-optical disk, or semiconductor memory. The removable recording medium 913 may be, for example, a DVD medium, an HD-DVD medium, or a Blu-ray (registered trademark) medium. The removable recording medium 913 may also be, for example, a CompactFlash (registered trademark) (CF), a flash memory, or an SD memory card (Secure Digital memory card). The removable recording medium 913 may also be, for example, an IC card (Integrated Circuit card) equipped with a contactless IC chip, or an electronic device.

[0166] The connection port 914 is a port for direct connection to the user terminal 10. Examples of the connection port 914 include a USB (Universal Serial Bus) port, an IEEE 1394 port, and a SCSI (Small Computer System Interface) port. Other examples of the connection port 914 include an RS-232C port, an optical audio terminal, and an HDMI (registered trademark) (High-Definition Multimedia Interface) port. By connecting an external device 915 to this connection port 914, the user terminal 10 can directly obtain various types of data from the external device 915 and provide various types of data to the external device 915.

[0167] The communication device 916 is, for example, a communication interface configured with a communication device or the like for connecting to a communication network 917. The communication device 916 is, for example, a communication card for a wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), or WUSB (Wireless USB). The communication device 916 may also be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication. The communication device 916 can transmit and receive signals, for example, between the Internet and other communication devices in accordance with a predetermined protocol such as TCP / IP. The communication network 917 connected to the communication device 916 is configured with a network connected by wire or wirelessly, and may be, for example, the Internet, a home LAN, infrared communication, radio wave communication, satellite communication, or the like.

[0168] The above describes an example of a hardware configuration capable of realizing the functions of the user terminal 10 constituting the information processing system 1 according to an embodiment of the present disclosure. Each of the above components may be configured using general-purpose components, or may be configured using hardware specialized for the function of each component. Therefore, the hardware configuration used can be changed as appropriate depending on the technical level at the time of implementing this embodiment. Note that, although not shown in FIG. 30, the information processing system 1 naturally includes various components corresponding to the user terminal 10.

[0169] It is possible to create a computer program for implementing each function of the user terminal 10 constituting the information processing system 1 according to this embodiment as described above and install it on a personal computer or the like. It is also possible to provide a computer-readable recording medium on which such a computer program is stored. Examples of the recording medium include a magnetic disk, an optical disk, a magneto-optical disk, and a flash memory. The computer program may also be distributed, for example, via a network without using a recording medium. The number of computers that execute the computer program is not particularly limited. For example, the computer program may be executed by multiple computers (e.g., multiple servers) in cooperation with each other.

[0170] <7. Supplementary Information> Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0171] For example, in the above embodiment, the user U1 flies the moving body 20 in advance and causes the imaging device 206 mounted on the moving body 20 to capture images for generating a virtual object, but the present technology is not limited to such an example. For example, if a virtual object has been generated by some method, such as if a virtual object has been generated based on an image captured in the past by a user other than the user U1 using the moving body 20, the already generated virtual object may be used.

[0172] Although the above embodiment describes the mobile object 20 as a drone, the mobile object 20 may be any mobile device. For example, the technology of the present disclosure may be applied to various types of flying objects, such as drones. Furthermore, the technology of the present disclosure may be applied to a manipulator corresponding to a robot's hand or arm. In this case, the information processing device may, for example, control the display of a virtual object to be handled by the manipulator on a display screen. Furthermore, the information processing device may generate movement information for controlling the movement of a mobile object, such as a fingertip of the manipulator, as the mobile object. This makes it possible to more intuitively generate movement information for controlling the movement of the manipulator's fingertip.

[0173] Furthermore, in the above embodiment, information relating to virtual objects, Waypoints, etc. is recorded in the information processing device 100. However, the present invention is not limited to this, and information relating to virtual objects, Waypoints, etc. may be recorded in various servers connected to a network. In this case, the information processing device 100 can receive information recorded in the server as appropriate via the network and generate movement information, imaging information, etc.

[0174] Furthermore, in the above embodiment, the user terminal 10 has been described assuming that it is mainly a smartphone or a tablet terminal, etc. However, the user terminal 10 is not limited to this, and may be a general-purpose personal computer (PC), a game console, a robot, or a wearable device such as an HMD (Head Mounted Display) or a smart watch.

[0175] Furthermore, the steps shown in the flowcharts of the above embodiments include not only processes performed in chronological order according to the order described, but also processes that are not necessarily performed in chronological order but are performed in parallel or individually. Needless to say, even for steps that are performed in chronological order, the order can be changed as appropriate in some cases.

[0176] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0177] The following configurations also fall within the technical scope of the present disclosure. (1) a display control unit that controls the display of a virtual object based on an object existing in real space on a display screen; a movement information generating unit that generates movement information for controlling the movement of the moving body; Information processing device. (2) the movement information generation unit generates the movement information based on an operation by a user viewing the display screen. The information processing device according to (1) above. (3) The display includes a route of the moving object. The information processing device according to (2) above. (4) At least a portion of the path is marked with one or more adjustment portions for adjusting the path; the operation is an operation of moving the position of the adjustment part displayed on the display screen, The information processing device according to (3) above. (5) the virtual object is displayed superimposed on an image captured by a first imaging device. The information processing device according to any one of (2) to (4) above. (6) the operation includes an operation of moving a viewpoint of the first imaging device, the movement information generation unit generates the movement information based on a movement of a predetermined position relative to the viewpoint. The information processing device according to (5) above. (7) the movement information generation unit generates the movement information based on an operation of moving a designated object that specifies a route of the moving body. The information processing device according to any one of (2) to (6) above. (8) the moving object includes a second imaging device that captures an image of a landscape; an imaging information generating unit that generates imaging information for controlling an imaging range of the second imaging device based on a user operation; The information processing device according to any one of (1) to (7) above. (9) the imaging information generation unit generates, as imaging information, direction information relating to a direction in which the second imaging device captures an image; The information processing device according to (8). (10) an image captured by a first imaging device is displayed on the display screen; the imaging information generation unit generates the direction information based on an operation of moving the orientation of the first imaging device. The information processing device according to (9) above. (11) the imaging information generation unit generates, as the imaging information, angle-of-view information for controlling an angle of view of the second imaging device, based on a pinch-out operation or a pinch-in operation on the display screen by a user. The information processing device according to any one of (8) to (10) above. (12) an imaging prediction unit that predicts an image to be captured by the second imaging device based on the movement information and the imaging information, The information processing device according to any one of (8) to (11) above. (13) a movement prediction unit that predicts movement of the moving object based on the movement information, The information processing device according to any one of (1) to (12) above. (14) The moving body is movable in three dimensions. The information processing device according to any one of (1) to (13). (15) The moving object is an aircraft. The information processing device according to (14) above. (16) The processor: Controlling the display of a virtual object based on an object existing in real space on a display screen; generating movement information for controlling movement of the moving object; Information processing methods. (17) On the computer, a function of controlling the display of virtual objects based on objects existing in real space on a display screen; a function of generating movement information for controlling the movement of a mobile object; A program to achieve this. [Explanation of symbols]

[0178] 10 User terminal 100 Information processing device 110 Imaging unit 120 Sensor unit 130 Input section 140 Acquisition Department 150 Processing section 151 Detection unit 157 Movement information generation unit 158 Imaging information generation unit 159 Display information generation section 161 Movement Prediction Unit 162 Imaging Prediction Unit 170 Display control unit 175 Display section 20 Mobile 202 aircraft 204 Propeller 206 Imaging device 402 Route 404 Virtual Route 406, 408, 410, 412 Waypoint 422, 432 Virtual Objects 610 display screen 612 images of virtual objects 614 Virtual Route Images Images from 616 Waypoint 622 Designated Objects

Claims

1. An imaging device mounted on a moving object captures an image of an object present in real space; generating scale information based on the scale of the captured real space by a processor; a display control step in which a processor displays, on a display screen, an object present in real space photographed by the imaging device and a virtual route along which the moving object will move, from a viewpoint of the imaging device while the moving object is moving, based on the generated scale information; having Information processing methods.

2. a virtual route generation step in which the processor generates a virtual route, which is a route along which the moving object will travel, in accordance with the scale of the real space based on an operation by a user viewing the display screen; a display control step in which a processor displays an object existing in real space and the generated virtual route on the display screen; the virtual route is displayed superimposed on an image captured by an imaging device mounted on the moving object; the operation includes an operation of moving a viewpoint of the imaging device; Information processing methods.

3. a movement information generating step in which a processor generates movement information adjusted based on an operation by a user viewing the display screen from the displayed virtual route; having 3. The information processing method according to claim 1 or 2.

4. a plane detection step in which a processor detects a plane based on information captured by the imaging device; having 3. The information processing method according to claim 1 or 2.

5. At least a portion of the virtual path includes one or more adjustment portions for adjusting the virtual path. is displayed, the operation is an operation of moving the position of the adjustment part displayed on the display screen, The information processing method according to claim 3 .

6. the virtual route is displayed superimposed on an image captured by the imaging device; The information processing method according to claim 1 .

7. In the virtual route generating step, the virtual route is generated based on an operation of moving a designated object that designates a route of the moving body. The information processing method according to claim 2 .

8. the moving body is capable of moving three-dimensionally based on the generated movement information; The information processing method according to claim 3 .

9. The moving object is an aircraft. The information processing method according to any one of claims 1 to 8.

10. An information processing system including a mobile object and an information processing device, The moving body is an imaging device that captures an image of an object existing in real space, The information processing device includes: a generation unit that generates scale information based on the scale of the captured real space; a display control unit that displays, on a display screen, an object present in real space photographed by the imaging device and a virtual route along which the moving object will move, from a viewpoint of the imaging device while the moving object is moving, based on the generated scale information; Equipped with Information processing system.

11. Computer, a generation unit that generates scale information based on the scale of the real space captured by an imaging device mounted on a moving body that captures an image of an object existing in the real space; a display control unit that displays, on a display screen, an object present in real space photographed by the imaging device and a virtual route along which the moving object moves, from a viewpoint of the imaging device while the moving object is moving, based on the generated scale information; A program to function as a

Images