Information processing methods, information processing systems, and programs

Abstract

To enable information for moving a moving object to be generated intuitively.SOLUTION: An information processing system is provided with: a display control unit for controlling display, on a display screen, of a virtual object based on an object being present in a real space; and a movement information generation unit for generating movement information for controlling the movement of a moving object. Thus, the information for moving the moving object can be generated intuitively.SELECTED DRAWING: Figure 2

Description

Technical Field

[0004] , , , , , , ,

[0006] , , , ,

[0005] , , , , ,

[0007] ,

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In recent years, mobile bodies such as drones controlled using a control device have been used. For example, a drone equipped with a camera is made to image a landscape from above, and the captured image is utilized.

[0003] For example, Patent Document 1 describes a technique for performing efficient image transfer by switching the mode from an image capturing mode to an image transfer mode when a predefined mode switching condition occurs.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] As a method for controlling the movement of a mobile body such as a drone, in addition to the method in which a user controls the movement of the mobile body using a control device, a method of presetting a path along which the mobile body moves in advance and moving the mobile body along the preset path can be considered. In this case, for example, it is conceivable to set the movement path of the mobile body on a map.

[0006] However, in the method of setting the path of the mobile body on a two-dimensional map, it has been difficult to intuitively set the movement of the mobile body when the mobile body moves three-dimensionally. The technique described in Patent Document 1 is not a technique intended to intuitively generate movement information for controlling the movement of a mobile body such as a drone.

[0007] Therefore, this disclosure proposes a novel and improved information processing device, information processing method, and program that can intuitively generate information for the movement of a moving object. [Means for solving the problem]

[0008] According to this disclosure, an information processing device is provided, comprising a display control unit that controls the display of virtual objects based on objects existing in real space on a display screen, and a movement information generation unit that generates movement information for controlling the movement of a moving object.

[0009] Furthermore, this disclosure provides an information processing method in which a processor controls the display of virtual objects based on real-world objects on a display screen, and generates movement information for controlling the movement of a moving object.

[0010] Furthermore, according to this disclosure, a program is provided for a computer that enables the display of virtual objects based on real-world objects on a display screen, and the generation of movement information for controlling the movement of a moving object. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows the configuration of an information processing system according to one embodiment of the present disclosure. [Figure 2] A functional block diagram showing the configuration of a user terminal according to one embodiment of this disclosure. [Figure 3] This is a functional block diagram showing the configuration of the processing unit. [Figure 4] This diagram shows a user operating a mobile device to take images of a tower and a forest. [Figure 5] This figure shows a virtual object generated based on the image of the tower. [Figure 6] This figure shows virtual objects generated based on images of a forest. [Figure 7] This diagram shows the path taken by the moving object. [Figure 8] This is a diagram showing a state where a flat surface on a desk existing in the real space is detected by a user terminal. [Figure 9] This is a diagram showing a state where a Waypoint is selected based on a user's operation. [Figure 10] This is a diagram showing a state where a Waypoint is selected based on a user's operation. [Figure 11] This is a diagram showing a state where the position of a Waypoint is adjusted based on a user's operation. [Figure 12] This is a diagram showing a state where the position of a Waypoint is adjusted based on a user's operation. [Figure 13] This is a diagram showing a state where a new path for a mobile body is set based on a user's operation. [Figure 14] This is a diagram showing a state where a new path for a mobile body is set based on a user's operation. [Figure 15] This is a diagram showing a state where the position of an imaging unit included in a user terminal is set as a Waypoint. [Figure 16] This is a diagram showing a display screen when the position of an imaging unit included in a user terminal is set as a Waypoint. [Figure 17] This is a diagram showing a state where a position at a predetermined distance away from a user terminal is set as a Waypoint. [Figure 18] This is a diagram showing a display screen when a position at a predetermined distance away from a user terminal is set as a Waypoint. <00000捌7>This is a diagram showing a state where a Waypoint is set using a pointing stick. [Figure 20] This is a diagram showing a display screen when a Waypoint is set by a pointing stick. [Figure 21] This is a diagram showing a state where the orientation of an imaging device of a mobile body is set by moving the orientation of a user terminal. [Figure 22]This is a diagram showing the state in which the angle of view of the imaging device of the moving body is set by performing a pinch operation on the display screen of the user terminal. [Figure 23] This is a diagram showing a display screen for displaying the simulation result of the movement of the moving body. [Figure 24] This is a diagram showing a display screen for displaying the simulation result of the image captured by the imaging device provided in the moving body. [Figure 25] This is a flowchart diagram showing an imaging method by manual operation. [Figure 26] This is a flowchart diagram showing a method of automatically flying the moving body and causing the imaging device to capture an image. [Figure 27] This is a flowchart diagram showing the procedure until a virtual object is generated. [Figure 28] This is a flowchart diagram showing the procedure until an image is captured based on the generated movement information and imaging information. [Figure 29] This is a diagram showing the display processing by the information processing device. [Figure 30] This is a functional block diagram showing an example of the hardware configuration of a user terminal that constitutes an information processing system according to an embodiment of the present disclosure.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, there are cases where a plurality of components having substantially the same functional configuration are distinguished by attaching different alphabets after the same reference numeral. For example, a plurality of configurations having substantially the same functional configuration may be distinguished as the user terminal 10a and the user terminal 10b as needed. However, when it is not necessary to particularly distinguish each of a plurality of components having substantially the same functional configuration, only the same reference numeral is attached. For example, when it is not necessary to particularly distinguish the user terminal 10a and the user terminal 10b, it is simply referred to as the user terminal 10.

[0013] The description will be made in the following order. 1. Structure 1.1. Configuration of the Information Processing System 1.2. User Terminal Configuration 2. Creating virtual objects 3. Example of Operation 3.1. Generation of movement information 3.2. Generation of imaging information 3.3. Simulation of the movement of a moving object 4. Imaging Method 4.1. Manual imaging method 4.2. Imaging Method by Automated Flight 4.3. Image capture method using a map displayed on the terminal's screen. 4.4. Imaging method relating to this disclosure 5. Effects 6. Hardware Configuration 7. Supplement

[0014] <1. Structure> <<1.1. Configuration of the Information Processing System>> First, with reference to Figure 1, the configuration of an information processing system 1 according to one embodiment of the present disclosure will be described. Figure 1 is a diagram showing the configuration of an information processing system 1 according to one embodiment of the present disclosure. The information processing system 1 comprises a user terminal 10 and a mobile device 20. The user terminal 10 and the mobile device 20 are connected to each other so as to be able to communicate.

[0015] The user terminal 10 may be, for example, a smartphone or a tablet device. The user terminal 10 generates movement information to control the movement of the mobile object 20 in response to user operations and transmits said movement information to the mobile object 20. The user terminal 10 can also display virtual objects, etc., as described later, in response to user operations.

[0016] The mobile device 20 is a device that moves based on movement information generated by the user terminal 10. Here, the mobile device 20 can be various types of mobile devices, but in the following description, the mobile device 20 will be described as a drone. The mobile device 20 may also be equipped with an imaging device for capturing images of the scenery.

[0017] <<1.2. User Terminal Configuration>> Referring to Figure 2, the configuration of a user terminal 10 according to one embodiment of this disclosure will be described. Figure 2 is a functional block diagram showing the configuration of a user terminal 10 according to one embodiment of this disclosure.

[0018] The user terminal 10 has the function of acquiring image information, sensor information, or information based on user operations, and outputting the results of various processing applied to the acquired information. The functions of the user terminal 10 are realized through the cooperation of the information processing device 100, imaging unit (first imaging device) 110, sensor unit 120, input unit 130, and display unit 175 equipped in the user terminal 10.

[0019] The imaging unit 110 may be any known imaging device for capturing images. The imaging unit 110 may have any known image sensor, such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. In addition to these image sensors, the imaging unit 110 may have any other components, such as a lens for forming an image of the subject on the image sensor, and a light source for illuminating the subject. The imaging unit 110 transmits the image information obtained from the image capture to the information processing device 100.

[0020] The sensor unit 120 includes at least one of various known sensors, such as a distance measuring sensor or an IMU (Inertial Measurement Unit). The distance measuring sensor may be, for example, a stereo camera or a ToF (Time of Flight) sensor. The distance measuring sensor detects distance information, such as the distance between the user terminal 10 and objects in its vicinity, and transmits the detected distance information to the information processing device 100. The IMU includes at least one of an acceleration sensor, a gyroscope, or a magnetic sensor. The IMU transmits the detected information as IMU information to the information processing device 100.

[0021] The input unit 130 has the function of generating input information based on user operations. The input unit 130 may be, for example, a touch panel. The input unit 130 generates input information based on various operations by the user, such as touch operations, drag operations, pinch-out operations, or pinch-in operations. The input unit 130 transmits the generated input information to the acquisition unit 140.

[0022] The information processing device 100 has the function of performing various processes based on the acquired information and controlling the display of the display unit 175 based on the results of said processing. The functions of the information processing device 100 are realized through the cooperation of the acquisition unit 140, processing unit 150, display control unit 170, storage unit 180, and communication control unit 190.

[0023] The acquisition unit 140 acquires information input from at least one of the imaging unit 110, the sensor unit 120, or the input unit 130. The acquisition unit 140 transmits the acquired information to the processing unit 150.

[0024] The processing unit 150 has the function of performing various processes based on the information transmitted from the acquisition unit 140. For example, the processing unit 150 has the function of generating information for controlling the mobile body 20 based on the information transmitted from the acquisition unit 140. The processing unit 150 also generates information related to the content to be displayed on the display screen of the display unit 175. The detailed configuration and functions of the processing unit 150 will be described later with reference to Figure 3. The processing unit 150 transmits the generated information to the display control unit 170, the storage unit 180, or the communication control unit 190.

[0025] The display control unit 170 has the function of controlling the display on the display screen of the display unit 175. For example, the display control unit 170 controls the display of virtual objects based on real-world objects on the display screen of the display unit 175 based on information transmitted from the processing unit 150.

[0026] The display unit 175 is a display device having the function of displaying various known images. In this embodiment, the display unit 175 and the aforementioned input unit 130 are integrated and configured as a touch panel. As will be described later, the user can cause the information processing device 100 to generate movement information for controlling the movement of the mobile body 20 by performing predetermined operations while referring to the display screen of the display unit 175.

[0027] The storage unit 180 has the function of storing various types of information, such as information generated or acquired by the information processing device 100. For example, the storage unit 180 may store information about a pre-generated virtual object. The method for generating a virtual object will be described later. The storage unit 180 may also store movement information for controlling the movement of the mobile body 20. More specifically, the storage unit 180 may store information (Waypoint information) about specific points (also referred to as "Waypoints") included in the path of the mobile body 20. The path of the mobile body 20 may be formed by connecting multiple Waypoints. The storage unit 180 may also store movement information or imaging information generated by the processing unit 150. The information stored in the storage unit 180 is referenced by the processing unit 150, the display control unit 170, or the communication control unit 190 as needed.

[0028] The communication control unit 190 has a function to control the transmission of various types of information generated by the processing unit 150. The communication control unit 190 controls the transmission of movement information or imaging information generated by the processing unit 150. This movement information or imaging information is transmitted to the mobile body 20. The mobile body 20 can move or take images based on the transmitted information.

[0029] Next, with reference to Figure 3, the processing unit 150 of the information processing device 100 will be described in more detail. Figure 3 is a functional block diagram showing the configuration of the processing unit 150. As shown in Figure 3, the processing unit 150 includes a detection unit 151, a self-position calculation unit 154, a virtual object calculation unit 155, a generation unit 156, and a prediction unit 160.

[0030] The detection unit 151 has the function of performing various detections based on the information transmitted from the acquisition unit 140. The functions of the detection unit 151 are realized by the plane detection unit 152 and the object detection unit 153. The plane detection unit 152 has the function of detecting planes included in an image based on image information and distance information, etc. The object detection unit 153 has the function of detecting predetermined objects included in an image based on image information and distance information, etc. The detection unit 151 transmits the detection results to the self-position calculation unit 154.

[0031] The self-position calculation unit 154 has the function of calculating the self-position of the user terminal 10. Here, the self-position of the user terminal 10 includes not only the position where the user terminal 10 is located, but also the orientation of the user terminal 10. Specifically, the self-position calculation unit 154 takes image information, distance information, and IMU information as input and calculates the position or orientation of the user terminal 10 relative to the environment or objects around the user terminal 10 using SLAM (Simultaneous Localization And Mapping) technology. At this time, the self-position calculation unit 154 may determine the origin or scale in SLAM based on object information or plane information, etc. The self-position calculation unit 154 transmits the calculated results to the virtual object calculation unit 155 and the generation unit 156.

[0032] The virtual object calculation unit 155 generates information about virtual objects to be placed on the display screen. More specifically, the virtual object calculation unit 155 calculates placement information (information about position or direction, etc.) or scale information for virtual objects to be placed on the display screen of the display unit 175, based on the self-position of the user terminal 10 calculated by the self-position calculation unit 154, the results detected by the detection unit 151, the input information input to the input unit 130, and the information about virtual objects stored in the storage unit 180. Here, the virtual object calculation unit 155 calculates the scale of the virtual object based on the scale of real space. More specifically, the virtual object calculation unit 155 determines the scale of the virtual object displayed on the display screen by appropriately scaling it from the scale of the real object on which the virtual object is based, and generates scale information. The virtual object calculation unit 155 transmits the calculated results to the movement information generation unit 157, which will be described later.

[0033] The generation unit 156 has the function of generating various types of information for controlling the mobile body 20. More specifically, the generation unit 156 generates information for controlling the movement of the mobile body 20, the operation of the imaging device (second imaging device) attached to the mobile body 20, and the display of the display unit 175. The functions of the generation unit 156 are realized by the movement information generation unit 157, imaging information generation unit 158, and display information generation unit 159, which are attached to the generation unit 156.

[0034] The movement information generation unit 157 generates movement information associated with a virtual object for controlling the movement of the moving object 20. Specifically, the movement information generation unit 157 generates the position and direction of a waypoint as movement information based on the user terminal 10's own position, the input information to the input unit 130, the virtual object's placement information, scale information, and waypoint information. For example, the movement information generation unit 157 may generate the path of the moving object 20 as movement information. In this case, the movement information generation unit 157 may generate movement information with a scale corresponding to the virtual object's scale information, or it may generate movement information by adjusting the said movement information to the scale of real space. Furthermore, the movement information generation unit 157 can also modify the movement information and generate new movement information based on user operations to the input unit 130, etc. Specific user operations will be described later. The movement information generation unit 157 transmits the generated movement information to the display information generation unit 159, the prediction unit 160, and the storage unit 180.

[0035] The imaging information generation unit 158 ​​has the function of generating imaging information to control the range captured by the imaging device equipped on the mobile body 20, based on user operation. More specifically, the imaging information generation unit 158 ​​generates imaging information related to the direction the imaging device faces or the field of view, etc., based on input information etc. generated by the input unit 130. The imaging information generation unit 158 ​​transmits the generated imaging information to the prediction unit 160 and the storage unit 180.

[0036] The display information generation unit 159 has the function of generating display information related to the content to be displayed on the display unit 175. More specifically, the display information generation unit 159 generates computer graphics (CG) to be displayed on the display unit 175 as display information, based on the position of the imaging unit 110, virtual object placement information, scale information, waypoint information, and prediction results from the prediction unit 160, which will be described later. The display information generation unit 159 can generate video showing the movement of the moving object 20 as display information. The display information generation unit 159 can also generate display information for displaying simulated video captured by the imaging device of the moving object 20. The display information generation unit 159 transmits the generated display information to the display control unit 170.

[0037] The prediction unit 160 has the function of predicting the movement of the moving object 20. More specifically, the prediction unit 160 can predict the movement of the moving object 20 and the operation of the imaging device attached to the moving object 20. The functions of the prediction unit 160 are realized by the movement prediction unit 161 and the imaging prediction unit 162. The prediction unit 160 transmits the predicted results to the display information generation unit 159.

[0038] The movement prediction unit 161 has the function of predicting the movement of the moving object 20 based on movement information. For example, the movement prediction unit 161 can predict the path of the moving object 20. The imaging prediction unit 162 predicts the image captured by the imaging device attached to the moving object 20 based on the movement information and imaging information. More specifically, the imaging prediction unit 162 predicts the image captured by the imaging device based on the path the imaging device will take and the orientation of the imaging device.

[0039] <2. Creating virtual objects> In this embodiment, a virtual object is displayed on the display unit 175, and the user can cause the information processing device 100 to generate movement information or imaging information, etc., by performing predetermined operations while viewing the display. Here, an example of a method for generating the virtual object is described. Note that the method for generating the virtual object is not limited to the method described below, and the virtual object may be generated by any method.

[0040] The method for generating virtual objects will be explained with reference to Figures 4 to 7. Figure 4 shows user U1 operating the mobile device 20 to image the tower 420 and the forest 430. Figure 5 shows a virtual object 422 generated based on the imaged tower 420. Figure 6 shows a virtual object 432 generated based on the imaged forest 430. Furthermore, Figure 7 shows the path 402 taken by the mobile device 20.

[0041] First, the method for generating virtual objects according to this embodiment will be briefly described. Here, user U1 wants the imaging device 206 to capture an image including the tower 420. In this embodiment, first, user U1 uses the control device 401 to operate the mobile body 20 and has the imaging device 206 capture images of the tower 420 and the surrounding forest 430 in real space. Then, based on the captured images, a three-dimensional virtual object is generated based on various CG technologies. In this embodiment, a waypoint is also set along the path taken by the mobile body 20, and movement information is generated. The method for generating virtual objects will be described in more detail below with reference to Figures 4 to 7.

[0042] First, as shown in Figure 4, user U1 uses the control device 401 to fly the mobile device (drone) 20. The mobile device 20 consists of a body 202, propellers 204, and an imaging device 206. The mobile device 20 can fly when the propellers 204 are driven. In addition, user U1 controls the direction, attitude, and speed of the body 202 by operating the propellers 204, etc. Furthermore, the imaging device 206 on the mobile device 20 captures images of the scenery around the mobile device 20. In this case, the imaging device 206 captures an image including the tower 420 and the surrounding forest 430 shown in Figure 4.

[0043] Based on the captured images, virtual objects of the tower 420 and the forest 430 are generated using various known computer graphics (CG) techniques. More specifically, a virtual object 422 of the tower 420 existing in real space as shown in Figure 5 and a virtual object 432 of the forest 430 existing in real space as shown in Figure 6 are generated. Information regarding these generated virtual objects is stored in the storage unit 180 of the information processing device 100.

[0044] Furthermore, user U1 may have the mobile body 20 perform the flight required to actually capture the desired image. More specifically, user U1 maneuvers the mobile body 20 so that it follows a path 402 that circles around the tower 420 shown in Figure 4. The mobile body 20 calculates the path 402 it has traveled using sensors on the mobile body 20 and records the calculated result on a recording medium or the like on the mobile body 20. Here, waypoints may be set in the path 402 according to a predetermined rule. For example, waypoints may be set at predetermined distance intervals. Also, the density of waypoints may be adjusted according to the curvature of the path 402.

[0045] Figure 7 shows an example of route 402 with Waypoint 406 set. While Figure 7 shows 13 Waypoints 406a to 406m, route 402 may have between 2 and 12 Waypoints, or 14 or more Waypoints. Furthermore, while Waypoints may be shown in the diagrams used in the following explanation, the number of Waypoints is not limited to those shown in those diagrams.

[0046] Information regarding the route 402 for which a Waypoint has been set in this manner may be stored as travel information in the storage unit 180 of the information processing device 100.

[0047] <3. Example of operation> Here, we will describe a specific example of an operation in which a user causes the information processing device 100 to generate movement information or imaging information, etc., based on the virtual object generated as described above.

[0048] <<3.1. Generation of movement information>> This section describes the case where movement information is generated based on user operations. First, referring to Figures 8 to 12, we will describe the case where movement information is previously recorded in the information processing device 100, and the existing movement information is modified to generate new movement information. Figure 8 shows the detection of a plane on the desk 500 in real space by the user terminal 10a. Figures 9 and 10 show the selection of Waypoint 408a based on the operations of user U2. Furthermore, Figures 11 and 12 show the adjustment of the position of Waypoint 408a based on the operations of user U2.

[0049] As shown in Figure 8, a desk 500 is assumed to exist in the real space in front of user U2. User U2 possesses a user terminal 10a, and the display screen 610 of the user terminal 10a displays a start button 602 for starting the display of virtual objects and setting a waypoint. Here, the display screen 610 also functions as an input unit 130 that accepts touch operations or pinch operations from user U2. When user U2 touches the start button 602, the user terminal 10a detects the plane 506 on the desk 500.

[0050] As shown in Figure 9, the display screen 610a of the user terminal 10a displays an image 612a of the pre-generated virtual object 422a of the tower and an image 614a of the virtual path 404a of the mobile body 20. Here, a virtual path is a path in the virtual space where the virtual object is placed. By appropriately scaling the virtual path, the actual path that the mobile body 20 actually travels is formed. Hereafter, when there is no particular distinction between a virtual path and an actual path, they will simply be referred to as "paths". At this time, the virtual object 422a is displayed on the display screen 610a as if it were placed on the desk 500. Note that although the virtual object 422a is shown on the desk 500 in Figure 9, the virtual object 422a is shown here for illustrative purposes only, and the virtual object 422a is not actually placed on the desk 500.

[0051] Here, the size of the image 614a of the virtual object 422a may be, for example, a size that fits on the desk 500 as shown in Figure 9. Also, the image 614a of the virtual path 404a displays the image 616a of Waypoint 408a for adjusting the virtual path 404a. In this embodiment, user U2 can also adjust the size of the image 612a of the virtual object displayed on the display screen 610a (i.e., the distance from the user terminal 10a to the virtual object 422a) by performing a pinch operation on the display screen 610a. At this time, the user terminal 10a may store the ratio between the size of the tower in the real space on which the virtual object 422a is based and the size of the virtual object 422a.

[0052] In this embodiment, since the image 612a of the virtual object 422a is displayed on a flat surface (on the desk 500), user U2 can perceive the virtual object 422a as being placed on the ground. As a result, user U2 can operate the user terminal 10a more intuitively.

[0053] Here, it is assumed that the mobile object 20 has already flown by manual control or the like, and that the virtual path 404a is generated based on that flight. Furthermore, it is assumed that waypoints have been set in advance for the virtual path 404a. Specifically, as shown in Figure 9, 13 waypoints, indicated by circles, have been set for the virtual path 404a. In addition, one of the 13 waypoints, Waypoint 408a, corresponds to the Waypoint image 616a displayed on the display screen 610a.

[0054] User U2 can select a Waypoint to be adjusted by touching the Waypoint image 616a displayed on the display screen 610a. Here, it is assumed that User U2 has selected Waypoint 408a on the virtual path 404a. When Waypoint 408a is selected, User U2 can adjust the position of Waypoint 408a by performing pinch or drag operations on the display screen 610a. In this embodiment, User U2 can also adjust the orientation of the imaging device of the mobile body 20 by operating the display screen 610a. For example, when Waypoint 408a is selected, User U2 can specify the orientation of the imaging device of the mobile body 20 when the mobile body 20 passes over the position corresponding to Waypoint 408a by performing pinch or drag operations on the display screen 610a.

[0055] Next, with Waypoint 408a selected, user U2 moves the position of user terminal 10a. For example, as shown in Figures 11 and 12, user U2 pulls user terminal 10a towards user U2. As a result, the position of the selected Waypoint 408a moves in accordance with the movement of user terminal 10b. Also, virtual path 404a changes to virtual path 404b corresponding to the position of Waypoint 408b after the movement. This change in virtual path 404 adjusts the actual path that the mobile body 20 will take. In this way, in this embodiment, the path of the mobile body 20 is adjusted based on the operation of user U2 to move user terminal 10a. In this manner, the path of the mobile body 20 is adjusted, and new movement information representing the adjusted path is generated. Based on this movement information, the mobile body 20 becomes capable of flying around tower 420.

[0056] This section describes the case where a waypoint is pre-configured for the virtual route 404a. If a waypoint is not pre-configured for the virtual route 404a, user U2 can also configure a waypoint by touching a portion of the image 614 of the virtual route 404 displayed on the display screen 610. Waypoints configured in this way can also be adjusted using the method described above.

[0057] Thus, according to this embodiment, user U2 can fine-tune the waypoint by operating on the display screen 610 and moving the user terminal 10a while the waypoint is set. This makes it possible to generate information for moving the mobile object 20 more intuitively.

[0058] The above describes a method for generating new movement information by adjusting the virtual path 404a of the mobile body 20 when the virtual path 404a of the mobile body 20 is pre-configured. Next, two methods for configuring the path of the mobile body 20 when the path of the mobile body 20 is not pre-configured will be described with reference to Figures 13 and 14. Figures 13 and 14 show how the path of the mobile body 20 is newly configured based on the operation of user U2.

[0059] In both methods, user U2 operates user terminal 10 to set the route and waypoints of the mobile object 20. Waypoints set by the method described with reference to Figures 13 and 14 may also be adjusted by the method described above with reference to Figures 8 to 12.

[0060] The first method for setting the path of the mobile object 20 will be explained with reference to Figure 13. First, user U2 touches a part of the display screen 610c of user terminal 10c (for example, a designated location 616c shown on the display screen 610c). While touching the designated location 616c, user U2 moves user terminal 10c downwards, for example, as shown by the dashed line. User terminal 10c stores the trajectory it has moved as the path of the mobile object 20. At this time, user terminal 10c may set a waypoint in the path and store it together with the path.

[0061] Next, a second method for setting the route of the mobile object 20 will be explained with reference to Figure 14. In the second method, user U2 sets Waypoint 408d by touching the display screen 610d of user terminal 10d, as shown in the upper part of Figure 14. The image 616d of the set Waypoint 408d is displayed on the display screen 610d.

[0062] Next, as shown in the lower part of Figure 14, user U2 moves the position of user terminal 10d and touches the display screen 610e of user terminal 10e at the new position to set Waypoint 408e. Thereafter, the movement of user terminal 10 and the setting of Waypoint 408 are repeated, and the route of the moving object 20 is generated by connecting the multiple set Waypoints 408. The generated route and Waypoints 408 are stored in user terminal 10.

[0063] In this way, according to this embodiment, even if the route of the mobile body 20 is not set in advance, the user U2 can set the route of the mobile body 20 by operating on the display screen 610 and operating the user terminal 10.

[0064] Here, the location of the Waypoint set by the operation of the user terminal 10 will be explained in more detail with reference to Figures 15 to 18. Figure 15 shows how the position of the imaging unit 110a of the user terminal 10f is set as Waypoint 410. Figure 16 shows the display screen 610f when the position of the imaging unit 110a of the user terminal 10f is set as Waypoint 410. Figure 17 shows how a position a predetermined distance away from the user terminal 10g is set as Waypoint 412. Furthermore, Figure 18 shows the display screen 610 when a position a predetermined distance away from the user terminal 10g is set as Waypoint 412.

[0065] First, the location of the Waypoint will be explained with reference to Figures 15 and 16. As shown in Figure 15, the virtual object 422a of the tower is placed on the desk 500. The imaging unit 110a of the user terminal 10f is imaging the area in front of the imaging unit 110a. In other words, the imaging unit 110a is imaging the area that includes the virtual object 422a. Here, the position of the imaging unit 110a is specified as the Waypoint.

[0066] At this time, as shown in Figure 16, the display screen 610f of the user terminal 10f displays an image 612f of a virtual tower object placed on the desk 500. The user can set a Waypoint by looking at the display screen 610f and touching the display screen 610f. The user can also move the user terminal 10f and set Waypoint 410 at a new location.

[0067] In this case, the image displayed on the display screen 610 may correspond to the image actually captured by the imaging device of the mobile body 20 at the Waypoint. In this case, the user can check in advance the image captured by the imaging device of the mobile body 20.

[0068] Next, with reference to Figures 17 and 18, a method for setting a Waypoint at a predetermined distance from the imaging unit 110a by the user terminal 10g will be described. Specifically, a position located at a distance d forward from the imaging unit 110a and slightly offset downward from the optical axis 414 of the imaging unit 110a is set as Waypoint 412.

[0069] At this time, as shown in Figure 18, the display screen 610g of the user terminal 10g displays an image 612g of the virtual object of the tower and an image 616 of the Waypoint. Furthermore, the display screen 610g displays a guide surface 617 connecting the user terminal 10g and the Waypoint image 616. The user can set Waypoint 412 by touching the display screen 610g while viewing the Waypoint image 616 placed on the guide surface 617.

[0070] In this case, since Waypoint 412 is located below the optical axis 414 of the imaging unit 110a, it is thought that the user can more easily recognize the position of Waypoint 412 by referring to the display screen 610g.

[0071] The above describes how to set a waypoint by operating the display screen 610. Next, variations in the method of setting a waypoint will be explained with reference to Figures 19 and 20. Specifically, the method of setting a waypoint using a designated object that specifies the path of the moving object 20 will be explained.

[0072] Figure 19 shows how a waypoint is set using the designated rod 620. Figure 20 shows the display screen 610h when a waypoint is set using the designated rod 620.

[0073] In this embodiment, a spherical object 622 is provided at the tip of the designating rod 620. Here, the designating rod 620 may be a stylus or the like that can be used to perform various operations by touching the user terminal 10. The user terminal 10h is equipped with a sensor that can detect the three-dimensional position of the object 622 on the designating rod 620. Specifically, the user terminal 10h is equipped with a distance measuring sensor such as a ToF sensor or a stereo camera. The user terminal 10 acquires position information of the object 622 based on the sensor information from the distance measuring sensor. The position information of the object 622 is expressed in three dimensions (x, y, z). Here, z is the direction of gravity (up and down direction). The x and y directions are perpendicular to the z direction and are orthogonal to each other.

[0074] Here, a Waypoint is set based on the location information. Specifically, for example, when a user performs a touch operation on the display screen 610h, the user terminal 10 sets the location of the designated object 622 as a Waypoint.

[0075] At this time, the display screen 610h of the user terminal 10h shows the image 618 of the designated rod and the image 616 of the designated object. Therefore, the user can set the waypoint while confirming the position of the designated object 622 on the display screen 610h.

[0076] <<3.2. Generation of Imaging Information>> The above describes the operations for generating movement information (more specifically, information including waypoints) to control the movement of the mobile body 20. Next, two methods for generating imaging information to control the range captured by the imaging device of the mobile body 20 will be described with reference to Figures 21 and 22. Figure 21 shows how the orientation of the imaging device of the mobile body 20 is set by moving the orientation of the user terminal 10i. Figure 22 shows how the field of view of the imaging device of the mobile body 20 is set by performing a pinch operation on the display screen of the user terminal 10j.

[0077] First, referring to Figure 21, we will explain how to set the direction in which the imaging device of the mobile body 20 takes images. For example, the user selects one of the waypoints included in the path of the mobile body 20. In this state, as shown in Figure 21, the user can adjust the direction in which the imaging unit 110a of the user terminal 10i takes images (i.e., the imaging ranges 134a and 134b) by moving the orientation of the user terminal 10i. Directional information regarding the direction in which the adjusted imaging unit 110a takes images is generated as imaging information. In other words, the user terminal 10i can generate directional information based on the orientation information of the user terminal 10i. Based on this directional information, the imaging device of the mobile body 20 can take images in the same direction as the adjusted imaging unit 110a at the set waypoint.

[0078] Furthermore, the user terminal 10 according to this embodiment can generate field-of-view information for controlling the field-of-view of the imaging device of the mobile body 20 based on pinch-out or pinch-in operations performed by the user on the display screen.

[0079] Next, with reference to Figure 22, a method for setting the field of view of the imaging device of the mobile body 20 will be described. For example, the user selects one of the waypoints included in the path of the mobile body 20. In this state, the user can adjust the imaging range 134 of the imaging unit 110a by performing a pinch-out or pinch-in operation on the display screen of the user terminal 10j, thereby setting the field of view of the imaging device when the mobile body 20 passes through the selected waypoint. In other words, the user terminal 10j can generate field of view information for controlling the field of view of the imaging device of the mobile body 20 based on the pinch-out or pinch-in operation performed by the user on the display screen.

[0080] The above describes the generation of direction information and field of view information by the user terminal 10 based on user operations. The imaging device of the mobile body 20 can perform imaging based on said direction information and field of view information. The above direction information and field of view information may be generated when the position of the waypoint is set, or they may be generated after the position of the waypoint is set.

[0081] <<3.3. Simulation of the movement of a mobile object>> Next, with reference to Figures 23 and 24, the simulation of the movement of the mobile object 20 by the user terminal 10 based on movement information and imaging information will be described. Specifically, the user terminal 10 simulates the movement of the mobile object 20 and the images captured by the imaging device attached to the mobile object 20. Figure 23 is a diagram showing a display screen 611 that displays the simulation results of the movement of the mobile object 20. Figure 24 is a diagram showing a display screen 610k that displays the simulation results of the images captured by the imaging device attached to the mobile object 20.

[0082] As shown in Figure 23, the display screen 611 of the user terminal 10 shows an image 612i of a virtual tower object placed on a desk and an image 630 of a moving object, symbolically represented by a triangle. When the simulation of the movement of the moving object image 630 starts, the moving object image 630 moves along a virtual path 615 connected by Waypoint images 616a to m. By observing the movement of the moving object image 630, the user can predict how the actual moving object 20 will move.

[0083] Furthermore, according to the user terminal 10k of this embodiment, it is also possible to simulate the images captured by the imaging device of the mobile body 20. Specifically, the user terminal 10k can display images that are predicted to be captured by the imaging device of the mobile body 20 when the mobile body 20 flies along the Waypoint set as described above. As shown in Figure 24, the images that are predicted to be captured are displayed on the display screen 610k of the user terminal 10k. By looking at the display screen 610k, the user can predict the images that will be captured by the imaging device of the mobile body 20. The user can also stop the video displayed on the display screen 610k by touching the stop button 619 shown in the center of the display screen 610k.

[0084] <4. Imaging Method> The following describes a method for capturing images of a landscape using the mobile device 20. First, three methods for capturing images of a landscape without using the technology described above will be described. Then, a method for capturing images of a landscape using the mobile device 20 with the technology described above will be described.

[0085] In the following explanation, it is assumed that the mobile device 20 (for example, a drone) is flown around a structure such as a tower to capture impressive images, for example, for commercial use. In such a case, the mobile device 20 needs to fly in three-dimensional space. Therefore, impressive images can only be captured by appropriately controlling various conditions such as the flight position, speed, and camera orientation of the mobile device 20. For this reason, advanced techniques are required to operate the mobile device 20 in order to capture impressive images.

[0086] Furthermore, it is difficult for a user to manually control the mobile device 20 to fly the same trajectory multiple times. Also, when imaging outdoors or when imaging a wide area, it is necessary to consider factors such as sunlight conditions and the movement of people, making the timing of imaging crucial. For this reason, by repeatedly imaging, images with favorable conditions can be captured.

[0087] <<4.1. Manual Imaging Method>> First, referring to Figure 25, we will explain how a user operates the mobile unit 20 manually using a control device and causes the mobile unit 20's imaging device to capture images of the scenery. Here, the control device is assumed to control the movement of the mobile unit 20 and the orientation of the imaging device it is equipped with. Figure 25 is a flowchart illustrating the manual imaging method. The manual imaging method will now be explained in accordance with the flowchart shown in Figure 25.

[0088] First, the user confirms the differences in images according to the imaging conditions (step S101). More specifically, the user actually flies the mobile body 20 around the building by manual operation and confirms the differences in how the images appear according to imaging conditions such as the orientation of the imaging device of the mobile body 20 or the distance between the mobile body 20 and the building. It is preferable that the user is someone who is familiar with operating the mobile body 20.

[0089] Next, the user uses the mobile device 20 to capture images (step S103). More specifically, the user manually controls the flight of the mobile device 20 and the orientation of the imaging device so that impressive images are captured, causing the imaging device of the mobile device 20 to capture images.

[0090] At this time, the user may display a two-dimensional map screen on various known mobile devices such as tablet terminals, along with the video captured by the imaging device installed on the mobile body 20, and display the flight path of the mobile body 20. Furthermore, the user may set waypoints along the said path based on predetermined rules. This allows the user to set waypoints while checking the captured video.

[0091] Next, the user checks the image captured in step S103 (step S105). If the flight of the mobile body 20 and the orientation of the imaging device are controlled as intended (step S107: YES), the user proceeds to step S109. On the other hand, if the flight of the mobile body 20 and the orientation of the imaging device are not controlled as intended (step S107: NO), the user returns to step S103.

[0092] Even if the flight of the mobile body 20 and the orientation of the imaging device are controlled as intended (step S107: YES), if the intended impressive image is not captured due to reasons such as the sun going down or an unintended person crossing in front of the imaging device (step S109: NO), the process returns to step S103. On the other hand, if the intended impressive image is captured (step S109: YES), the imaging method shown in Figure 25 is terminated.

[0093] The above describes a method for capturing images using manual operation. With this method, in order to obtain the desired image, it is necessary to manually reproduce the same flight path of the mobile body 20 and the orientation of the imaging device multiple times. Therefore, obtaining the desired image is time-consuming and requires manual intervention each time an image is captured.

[0094] <<4.2. Imaging Method by Automated Flight>> Next, referring to Figure 26, we will explain how to have the mobile body 20 fly automatically while the imaging device captures images. Figure 26 is a flowchart illustrating how to have the mobile body 20 fly automatically while the imaging device captures images. The following explanation will follow the flowchart shown in Figure 26.

[0095] First, the processes in steps S201 to S207 are executed. However, since the processes in steps S201 to S207 are essentially the same as those in steps S101 to S107, we will omit their explanation here.

[0096] If the flight of the mobile body 20 and the orientation of the imaging device are controlled as intended (step S207: YES), imaging condition data is saved (step S209). More specifically, when the flight of the mobile body 20 and the orientation of the imaging device are controlled as intended, various imaging conditions such as the position, speed, and orientation of the imaging device of the mobile body 20 during flight are recorded. The imaging conditions are recorded on various known recording media equipped on the mobile body 20, etc. Information regarding the position or speed of the mobile body 20 is acquired by the GPS or IMU equipped on the mobile body 20, etc.

[0097] Next, the imaging operation is reproduced (step S211). More specifically, the flight of the mobile body 20 and the orientation of the imaging device are automatically reproduced based on the imaging conditions recorded in step S209. The image captured at this time is then reviewed by the user.

[0098] If the intended image is not captured (step S213: NO), the process returns to step S211 and the imaging operation is repeated. On the other hand, if, for example, the sunlight conditions are met and the intended image is captured (step S213: YES), the imaging shown in Figure 26 is completed.

[0099] The above describes the imaging method using automated flight. With this method, the user does not need to manually operate the same mobile device 20, thus reducing the burden on the user.

[0100] Furthermore, based on the data recorded in step S209, it is possible to display a map on the screen of a tablet device, for example, to depict the virtual route of the mobile object 20 on the map, and to fine-tune the virtual route. However, when the virtual route is displayed on a two-dimensional map screen, it is difficult to intuitively adjust, for example, the altitude of the virtual route.

[0101] Furthermore, since the self-position of the mobile object 20 is calculated using GPS (Global Positioning System) or IMU, there will be an error of approximately 50 cm to 1 m in the relative position to buildings, etc., which depends on GPS.

[0102] <<4.3. Image capture method using a map displayed on the terminal's screen>> The method described above requires the user to go to the actual location where the mobile device 20 will fly and to operate the mobile device 20 and set waypoints. Therefore, a possible method is to pre-specify the flight path of the mobile device 20 using a map displayed on the screen of a tablet device or similar, and then have the mobile device 20 fly along that specified path.

[0103] More specifically, the user sets a Waypoint by, for example, displaying a map on the screen of a tablet device and touching the screen. The location of the Waypoint may be set using longitude and latitude. At this time, the user may also set the speed and altitude of the mobile object 20 at the designated Waypoint. Furthermore, the user can also set the orientation of the mobile object 20 at the designated Waypoint. For example, the orientation of the mobile object 20 can be set to face the direction of travel.

[0104] Similarly, multiple waypoints are set, and by connecting these waypoints, a route for the mobile object 20 is established. The user can record or transmit information representing the set route to the mobile object 20, thereby causing the mobile object 20 to fly along the set route and allowing the imaging device to capture images.

[0105] According to this method, the user can set the flight path of the mobile device 20 before going to the site where the mobile device 20 will fly. Therefore, the user can set the flight path of the mobile device 20 while remaining at their company or home. However, it is not possible to know what kind of images will be captured by the imaging device of the mobile device 20 until the mobile device 20 is actually flown and images are captured by the imaging device. Furthermore, it is not possible to know how the images captured by the imaging device will change if the position of the waypoint is corrected until the mobile device 20 is actually flown and images are captured by the imaging device.

[0106] Furthermore, the method of setting waypoints by touching the map displayed on the screen is convenient when the mobile body 20 is to fly roughly over a wide area. However, when the imaging device of the mobile body 20 is to dynamically capture images around a building, it is considered difficult to precisely set the path of the mobile body 20. In addition, since a two-dimensional map is displayed on the screen, the altitude of the mobile body 20 must be set numerically, making it impossible to set waypoints intuitively.

[0107] <<4.4. Imaging Method Related to This Disclosure>> Next, the imaging method related to this disclosure will be described with reference to Figures 27 to 29. Figure 27 is a flowchart showing the steps until a virtual object is generated. Figure 28 is a flowchart showing the steps until an image is captured based on the generated movement information and imaging information. Furthermore, Figure 29 is a diagram showing the display processing by the information processing device 100. The imaging method related to this disclosure will be described below with reference to Figures 27 to 29. In the following description, Figures 2 to 24 mentioned above will be referenced as appropriate.

[0108] The user operation in step S301 shown in Figure 27 is substantially the same as the user operation in step S101. However, in the imaging method according to this disclosure, the user operating the mobile body 20 may be unfamiliar with operating the mobile body 20.

[0109] Next, the user causes the imaging device of the mobile body 20 to capture an image (step S303). The user causes the imaging device of the mobile body 20 to capture an image of the subject that forms the basis of the virtual object. At this time, the user may manually control the flight of the mobile body 20 and the orientation of the imaging device so that an impressive image is captured, and cause the imaging device of the mobile body 20 to capture an image. For example, the user may cause the mobile body 20 to orbit around the tower 420 as shown in Figure 4, and cause the imaging device 206 to capture an image. At this time, the imaging device 206 shall capture an image that includes the tower 420 and the forest 430.

[0110] Next, the user checks the captured image (step S305). More specifically, the user confirms that the image captured by the imaging device 206 is the intended image.

[0111] Next, virtual objects are generated (step S307). More specifically, based on the image captured in step S303, the position and orientation of the moving object 20 and the orientation of the imaging device 206 at the time the image was captured, information about three-dimensional virtual objects is generated using various known CG techniques. For example, information about the virtual tower object 422 shown in Figure 5 and the virtual forest object 432 shown in Figure 6 is generated.

[0112] Furthermore, the position and orientation of the mobile object 20 may be calculated more accurately not only using GPS and IMU, but also through Bundle Adjustment, a process that generates a 3D virtual object based on the captured image. This allows for a more accurate calculation of the relative position or orientation of the mobile object 20 with respect to the environment, such as buildings. Here, Bundle Adjustment is a method for estimating various parameters from an image with high accuracy. Information regarding the virtual object generated at this time is recorded in the storage unit 180 of the user terminal 10. At this time, information regarding the path 402 and waypoint 406 traveled by the mobile object 20, as shown in Figure 7, may also be recorded in the storage unit 180.

[0113] The process up to the creation of a virtual object has been described above with reference to Figure 27. Next, the procedure for capturing the desired image will be described with reference to Figures 28 and 29. Note that the processing in steps S401 to S405 shown in Figures 28 and 29 is mainly performed by the information processing device 100 according to one embodiment of this disclosure.

[0114] The information processing device 100 performs the virtual object display process (step S401). The virtual object display process will be explained with reference to Figure 29. Figure 29 is a flowchart illustrating the virtual object display process. The virtual object display process will be explained below in accordance with the flowchart shown in Figure 29. The process shown in Figure 29 is executed, for example, when the start button 602 displayed on the display screen 610 of the user terminal 10a is touched, as explained with reference to Figure 8.

[0115] First, the acquisition unit 140 acquires image information and sensor information (step S501). More specifically, the acquisition unit 140 acquires image information including the desk 500 captured by the imaging unit 110. The acquisition unit 140 also acquires IMU information detected by the sensor unit 120 or distance information from the user terminal 10a to the desk 500. The acquisition unit 140 transmits the acquired image information and distance information to the detection unit 151 of the processing unit 150. The acquisition unit 140 also transmits the acquired image information, distance information and IMU information to the self-position calculation unit 154 of the processing unit 150.

[0116] Next, the plane detection unit 152 detects a plane based on the image information and distance information transmitted from the acquisition unit 140 (step S503). Here, the plane detection unit 152 detects a flat plane 506 on the desk 500. The plane detection unit 152 transmits the detected result to the virtual object calculation unit 155.

[0117] Next, the self-position calculation unit 154 calculates the self-position of the user terminal 10 based on image information, distance information, and IMU information (step S505). More specifically, the self-position calculation unit 154 calculates the position and orientation of the user terminal 10 relative to the desk 500 or the surrounding environment. The self-position calculation unit 154 transmits the calculated results to the virtual object calculation unit 155.

[0118] Next, the virtual object calculation unit 155 calculates the position, orientation, and scale of the virtual object to be placed based on the calculation results of the self-position calculation unit 154 and the information about the virtual object recorded in the storage unit 180 (step S507). The virtual object calculation unit 155 transmits the calculated results to the movement information generation unit 157.

[0119] Next, the movement information generation unit 157 sets the path of the moving object 20 based on the calculation results of the virtual object calculation unit 155 and the waypoint information recorded in the storage unit 180 (step S509). For example, the movement information generation unit 157 sets a virtual path that revolves around a virtual object placed on the desk 500. The movement information generation unit 157 transmits information about the set virtual path to the display information generation unit 159.

[0120] Next, the display information generation unit 159 generates display information (step S511). More specifically, the display information generation unit 159 generates display information that shows the virtual path of the moving body 20 around a virtual object placed on the desk 500, and transmits the generated display information to the display control unit 170.

[0121] Next, the display control unit 170 controls the display of the display unit 175 so that an image of a virtual path is displayed around a virtual object placed on the desk 500 (step S513). As a result, the display screen of the display unit 175 displays an image 612 of the virtual tower object 422 on the desk 500 in front of the user, and an image 614 of the virtual path revolving around it.

[0122] The display process of virtual objects has been explained above with reference to Figure 29. Next, we will return to Figure 28 and explain the imaging method related to this disclosure.

[0123] The information processing device 100 generates movement information and imaging information (step S403). For example, as explained with reference to Figures 9 to 22, movement information such as waypoints and imaging information such as the orientation of the imaging device and zoom level are generated based on operations such as the user moving the user terminal 10, and transmitted to the prediction unit 160. Now, the processing of the information processing device 100 in the operations explained with reference to Figures 9 to 22 will be described.

[0124] (Process to adjust pre-set waypoints) Here, we will explain the processing of the information processing device 100 in the process in which the Waypoint is adjusted, as explained with reference to Figures 9 to 12. As shown in Figure 9, when user U2 touches the Waypoint image 616a displayed on the display screen 610a, the input unit 130 transmits input information to the movement information generation unit 157 indicating that the Waypoint 408a corresponding to the Waypoint image 616a has been selected.

[0125] Next, as shown in Figures 11 and 12, when user U2 pulls user terminal 10 towards user U2, the sensor unit 120 detects the movement of user terminal 10 and transmits the detected sensor information to the self-position calculation unit 154. Based on the sensor information, the self-position calculation unit 154 calculates the position and orientation of user terminal 10. The self-position calculation unit 154 then transmits the calculated results to the movement information generation unit 157.

[0126] Next, the movement information generation unit 157 modifies the virtual path of the moving body 20 so that the position of the selected Waypoint 408a is displaced by the distance the user terminal 10 has moved. As a result, information regarding the new virtual path is generated as movement information and transmitted to the prediction unit 160.

[0127] (Process to set a new Waypoint) Next, the processing performed by the information processing device 100 in the operation described with reference to Figure 13 or Figure 14 will be explained. The acquisition unit 140 acquires input information from the input unit 130 based on the user's operation on the display screen 610. The acquisition unit 140 also acquires image information from the imaging unit 110, distance information and IMU information from the sensor unit 120. The acquisition unit 140 transmits the acquired information to the processing unit 150.

[0128] The self-position calculation unit 154 calculates the self-position of the user terminal 10 based on the transmitted sensor information or distance information, and transmits the calculated result to the generation unit 156. The movement information generation unit 157 identifies the location of the waypoint based on the calculated result and input information, etc. The movement information generation unit 157 sets a virtual path for the moving object 20 by connecting the identified waypoints, and transmits the virtual path as movement information to the prediction unit 160.

[0129] (Process to set a Waypoint using a designated rod) Next, the processing of the information processing device 100 when setting a waypoint using the designated rod 620, as explained with reference to Figures 19 and 20, will be described. First, the acquisition unit 140 acquires IMU information and sensor information from the sensor unit 120. Furthermore, the acquisition unit 140 acquires image information from the imaging unit 110 and transmits this image information to the object detection unit 153.

[0130] The object detection unit 153 detects the designated object 622 included in the image based on the image information. Furthermore, the object detection unit 153 detects, for example, the distance and direction from the imaging unit 110 to the designated object 622 based on the sensor information, and transmits the detected results to the generation unit 156.

[0131] The movement information generation unit 157 identifies the location of the designated object 622 based on the results detected by the object detection unit 153 and sets that location as a waypoint. The movement information generation unit 157 sets a virtual path by connecting the set waypoints and transmits this virtual path as movement information to the prediction unit 160.

[0132] (Process to set the orientation of the imaging device) Next, the processing of the information processing device 100 when setting the direction of the imaging device, as explained with reference to Figure 21, will be described. The self-position calculation unit 154 calculates the orientation of the user terminal 10 based on the sensor information and transmits the calculated result to the generation unit 156. The imaging information generation unit 158 ​​sets the direction of the imaging device based on the calculated orientation. The imaging information generation unit 158 ​​transmits the set direction of the imaging device as direction information to the prediction unit 160.

[0133] (Process to set the field of view of the imaging device) Next, we will explain the processing of the information processing device 100 when setting the field of view of the imaging device, as described with reference to Figure 22.

[0134] The imaging information generation unit 158 ​​acquires input information from the input unit 130 indicating that a pinch-in or pinch-out operation has been performed by the user. Based on the input information, the imaging information generation unit 158 ​​generates field-of-view information representing the field of view of the imaging device and transmits it to the prediction unit 160.

[0135] Next, the information processing device 100 performs a simulation of the movement of the moving object 20 and the image captured by the imaging device (step S405). For example, the prediction unit 160 simulates the movement of the moving object 20 on the display screen 611. Specifically, the movement prediction unit 161 predicts the movement of the moving object 20 based on the movement information and transmits the prediction result to the display information generation unit 159.

[0136] Furthermore, the prediction unit 160 simulates the image to be captured. More specifically, the imaging prediction unit 162 predicts the image captured by the moving object 20 based on the movement information and imaging information, and transmits the prediction result to the display information generation unit 159.

[0137] Next, the display unit 175 displays the prediction result (step S407). More specifically, the display information generation unit 159 generates display information for displaying the prediction result based on the prediction result and transmits it to the display control unit 170. Based on the display information, the display control unit 170 controls the display of the display unit 175 so that the display unit 175 displays the prediction result. As a result, the prediction result is displayed on the display unit 175. More specifically, the prediction result of the movement of the moving body 20 shown in Figure 23 or the prediction result of the image captured by the imaging device of the moving body 20 shown in Figure 24 is displayed.

[0138] Next, if the simulation is performed as intended (step S409: YES), the process proceeds to step S411. At this time, the memory unit 180 may store the movement information and imaging information used in the simulation. On the other hand, if the simulation is not performed as intended (step S409: NO), the process returns to step S403.

[0139] Next, the mobile body 20 moves and the imaging device takes images (step S411). For example, the virtual path of the mobile body 20 formed in step S403 is converted into a real-space coordinate system by the movement information generation unit 157, thereby converting it into the actual path that the mobile body 20 actually travels. Information regarding this actual path is transmitted to the mobile body 20 by the communication control unit 190. As the mobile body 20 flies along the generated actual path, the imaging device takes images of the scenery based on the imaging information.

[0140] If an impressive image as intended is captured (Step S413: YES), the imaging process shown in Figure 28 is terminated. On the other hand, if an impressive image as intended is not captured (Step S413: NO), the process returns to Step S411.

[0141] <5. Effects> The imaging method relating to this disclosure has been described above. The information processing device 100 relating to this disclosure controls the display of virtual objects based on real-world objects on the display screen and generates movement information for controlling the movement of a mobile object. Therefore, if a user wants to have a mobile object 20, such as a drone, fly around a real-world object that is the basis of the virtual object, the user can specify the path of the mobile object 20 while viewing the virtual object. Accordingly, the information processing device 100 according to this embodiment makes it possible to generate movement information for controlling the movement of the mobile object 20 in a more intuitive manner.

[0142] Furthermore, according to the information processing device 100 of this embodiment, the movement information generation unit 157 generates movement information based on operations performed by user U2 viewing the display screen 610. User U2 can specify movement information such as the path of the moving object 20 while viewing the virtual object displayed on the display screen 610. This makes it possible to generate movement information for controlling the movement of the moving object 20 in a more intuitive manner.

[0143] Furthermore, in this embodiment, as shown in Figures 9 and 11, the display screen 610 includes an image 614 of the virtual path of the mobile object 20. This makes it easier for user U2 to visualize the path of the mobile object 20.

[0144] Furthermore, in this embodiment, at least a portion of the image 614 of the virtual route displayed on the display screen 610 displays one or more Waypoint (adjustment portion) images 616 for adjusting the route of the mobile body 20. The movement information generation unit 157 generates movement information based on the operation of moving the Waypoint images 616. Therefore, the user U2 can specify the route of the mobile body 20 simply by moving the Waypoint images 616 that serve as route markers, making it possible to generate movement information for controlling the movement of the mobile body 20 in a more intuitive manner.

[0145] Furthermore, in this embodiment, as explained with reference to Figure 13, the movement information generation unit 157 moves the position of the Waypoint 408 based on the operation of moving the position of the display screen 610. Therefore, the user can specify the path of the moving object 20 more intuitively.

[0146] Furthermore, in this embodiment, the image 612 of the virtual object is displayed superimposed on the image captured by the imaging unit 110 of the user terminal 10. As a result, user U2 can perceive the virtual object 422a as if it existed in real space. This allows user U2 to specify the path of the moving object 20 more intuitively. In addition, by aligning the viewpoint of the imaging unit 110 with the waypoint, the image captured from the waypoint can also be displayed on the display screen. As a result, it is possible to predict in advance how the captured image will change when the position of the waypoint changes.

[0147] Furthermore, in this embodiment, as explained with reference to Figures 15 to 18, the movement information generation unit 157 generates movement information based on the user U2's operation to move the viewpoint of the imaging unit 110. More specifically, the movement information generation unit 157 generates movement information based on the movement of a predetermined position relative to the viewpoint. The display screen 610 includes images captured by the imaging unit 110. Therefore, when the moving body 20 actually moves, the user U2 can easily visualize the scenery captured by the imaging device of the moving body 20, making it possible to have the imaging device of the moving body 20 capture more desired images.

[0148] Furthermore, the path of the moving object 20 may be the viewpoint of the imaging unit 110, as explained with reference to Figures 15 and 16. Alternatively, the path of the moving object 20 may be a position located a predetermined distance away from the viewpoint of the imaging unit 110. For example, as explained with reference to Figure 17, the path of the moving object 20 may be specified as a position located a distance d forward from the viewpoint of the imaging unit 110 and lowered to a degree that it fits within the field of view of the optical axis of the imaging unit 110. In this case, as shown in Figure 18, a Waypoint or the like will be displayed on the display screen 610g, allowing the user U2 to specify the path of the moving object 20 more intuitively.

[0149] Furthermore, in this embodiment, as described with reference to Figures 19 and 20, the movement information generation unit 157 generates movement information based on an operation to move a designated object that specifies the path of the moving body 20. More specifically, in this embodiment, the path of the moving body 20 is specified based on the path taken by the designated object 622, which is located at the tip of the designated rod 620. This allows the user to specify the path of the moving body 20 with the simple operation of moving the designated object 622. Also, as shown in Figure 20, an image 616 of the designated object 622 is displayed on the display screen 610h. Therefore, the user U2 can recognize the position of the designated object 622 via the display screen 610h. As a result, the user U2 can better visualize the path of the moving body 20.

[0150] Furthermore, in this embodiment, the mobile body 20 is equipped with an imaging device. This imaging device captures the scenery around the mobile body 20. In addition, the information processing device 100 according to this embodiment includes an imaging information generation unit 158 ​​that generates imaging information to control the range captured by the imaging device of the mobile body 20 based on user operations. Therefore, the user can specify the range captured by the imaging device of the mobile body 20 based on various operations, thereby causing the imaging device of the mobile body 20 to capture more appropriate images.

[0151] Furthermore, in this embodiment, the imaging information generation unit 158 ​​generates directional information as imaging information regarding the direction in which the imaging device of the mobile body 20 takes images. Therefore, the user can cause the imaging device of the mobile body 20 to capture more appropriate images.

[0152] Furthermore, in this embodiment, the display screen shows the image captured by the imaging unit 110. Also, as explained with reference to Figure 21, the imaging information generation unit 158 ​​generates direction information based on the operation of moving the orientation of the imaging unit 110. Therefore, the user can obtain direction information while inferring the image captured by the imaging device of the moving body 20, thereby causing the imaging device to capture a more appropriate image.

[0153] Furthermore, in this embodiment, the imaging information generation unit 158 ​​can generate field-of-view information as imaging information for controlling the field-of-view of the imaging device of the mobile body 20 based on a pinch-out or pinch-in operation on the display screen by the user U2. Therefore, the user can easily specify the range to be imaged by the imaging device of the mobile body 20.

[0154] Furthermore, in this embodiment, the information processing device 100 further includes a movement prediction unit 161 that predicts the movement of the moving object 20 based on movement information. More specifically, the movement prediction unit 161 can simulate the movement of the moving object 20. Therefore, the user can check the path of the moving object 20 in advance based on the simulation of its movement. In this embodiment, as explained with reference to Figure 23, the simulation result (i.e., prediction result) of the movement of the moving object 20 is displayed on the display screen 611. Therefore, the user can more easily check the path of the moving object 20 by looking at the display screen 611.

[0155] Furthermore, in this embodiment, the information processing device 100 further includes an imaging prediction unit 162 that predicts the image to be captured by the imaging device of the moving body 20 based on movement information and imaging information. In this embodiment, the imaging prediction unit 162 can simulate the image to be captured by the imaging device. The user can confirm the captured image based on the results of the simulation. Also, in this embodiment, as explained with reference to Figure 24, the results of the simulation by the imaging prediction unit 162 are displayed on the display screen 610k. Therefore, the user can easily confirm the prediction results by the imaging prediction unit 162.

[0156] Furthermore, in this embodiment, the mobile body 20 is capable of moving in three dimensions. Therefore, the user can specify the path of the mobile body 20 in three dimensions. As a result, the user can generate movement information to control the movement of the mobile body 20 more intuitively.

[0157] Furthermore, according to this embodiment, once the path of the mobile body 20 is set, it is possible to have the mobile body 20 fly the same path repeatedly without human intervention, or to have the imaging device capture similar images repeatedly.

[0158] <6. Hardware Configuration> Next, with reference to Figure 30, an example of the hardware configuration of a user terminal 10 constituting the information processing system 1 according to one embodiment of this disclosure will be described in detail. Figure 30 is a functional block diagram showing an example of the hardware configuration of a user terminal 10 constituting the information processing system 1 according to one embodiment of this disclosure.

[0159] The user terminal 10 constituting the information processing system 1 according to this embodiment mainly comprises a CPU 901, a ROM 902, and a RAM 903. The user terminal 10 also further comprises a host bus 904, a bridge 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device 910, a drive 912, a connection port 914, and a communication device 916.

[0160] The CPU 901 functions as an arithmetic processing unit and control unit, controlling all or part of the operations within the user terminal 10 according to various programs recorded in the ROM 902, RAM 903, storage device 910, or removable recording medium 913. The ROM 902 stores programs and arithmetic parameters used by the CPU 901. The RAM 903 temporarily stores programs used by the CPU 901 and parameters that change as appropriate during program execution. These are interconnected by a host bus 904, which is composed of an internal bus such as a CPU bus. For example, the acquisition unit 140, processing unit 150 (each functional unit shown in Figure 3), display control unit 170, and communication control unit 190 shown in Figure 2 may all be composed of the CPU 901.

[0161] The host bus 904 is connected to an external bus 906, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 905. The external bus 906 is also connected to an input device 908, an output device 909, a storage device 910, a drive 912, a connection port 914, and a communication device 916 via an interface 907.

[0162] The input device 908 is, for example, a user-operated operating means such as a mouse, keyboard, touch panel, button, switch, lever, and pedal. Alternatively, the input device 908 may be, for example, a remote control using infrared or other radio waves (a so-called remote control), or an external connected device 915 such as a mobile phone or PDA that is compatible with the operation of the user terminal 10. Furthermore, the input device 908 is comprised of, for example, an input control circuit that generates an input signal based on information input by the user using the above-mentioned operating means and outputs it to the CPU 901. The user of the user terminal 10 can input various data or instruct processing operations to the user terminal 10 by operating this input device 908.

[0163] The output device 909 is comprised of a device capable of visually or audibly notifying the user of acquired information. Such devices include display devices such as CRT displays, liquid crystal displays, plasma displays, EL displays, and lamps, as well as audio output devices such as speakers and headphones, and printers. The output device 909 outputs, for example, the results obtained from various processes performed by the user terminal 10. Specifically, the display device displays the results obtained from various processes performed by the user terminal 10 in text or images. On the other hand, the audio output device converts the audio signal, consisting of played-back audio data or sound data, into an analog signal and outputs it.

[0164] The storage device 910 is a data storage device configured as an example of the storage unit of the user terminal 10. The storage device 910 is composed of, for example, a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. This storage device 910 stores programs executed by the CPU 901 and various data. For example, the storage unit 180 shown in Figure 2 may be composed of the storage device 910.

[0165] The drive 912 is a reader / writer for recording media and is either built into or external to the user terminal 10. The drive 912 reads information recorded on the removable recording media 913, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs it to the RAM 903. The drive 912 can also write data to the removable recording media 913, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory. The removable recording media 913 is, for example, DVD media, HD-DVD media, or Blu-ray® media. The removable recording media 913 may also be CompactFlash® (CF), flash memory, or SD memory card (Secure Digital memory card). The removable recording media 913 may also be, for example, an IC card (Integrated Circuit card) or electronic device equipped with a contactless IC chip.

[0166] The connection port 914 is a port for direct connection to the user terminal 10. Examples of connection port 914 include USB (Universal Serial Bus) ports, IEEE1394 ports, and SCSI (Small Computer System Interface) ports. Other examples of connection port 914 include RS-232C ports, optical audio terminals, and HDMI (High-Definition Multimedia Interface) ports. By connecting an external device 915 to this connection port 914, the user terminal 10 can directly acquire various data from the external device 915 or provide various data to the external device 915.

[0167] The communication device 916 is a communication interface composed of, for example, a communication device for connecting to a communication network 917. The communication device 916 is, for example, a communication card for wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), or WUSB (Wireless USB). Alternatively, the communication device 916 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication. This communication device 916 can, for example, send and receive signals to and from the Internet or other communication devices in accordance with a predetermined protocol such as TCP / IP. The communication network 917 connected to the communication device 916 is composed of a network connected by wire or wireless, and may be, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

[0168] The above describes an example of a hardware configuration capable of realizing the functions of a user terminal 10 constituting an information processing system 1 according to one embodiment of this disclosure. Each of the above components may be made up of general-purpose materials, or it may be made up of hardware specialized for the functions of each component. Therefore, it is possible to change the hardware configuration used as appropriate depending on the level of technology at the time of implementing this embodiment. Although not shown in Figure 30, various configurations corresponding to the user terminal 10 constituting the information processing system 1 are naturally included.

[0169] Furthermore, it is possible to create computer programs to realize each function of the user terminal 10 that constitutes the information processing system 1 according to this embodiment as described above, and to implement them on a personal computer or the like. A computer-readable recording medium on which such computer programs are stored can also be provided. Examples of recording media include magnetic disks, optical disks, magneto-optical disks, and flash memory. Alternatively, the computer programs may be distributed without using a recording medium, for example, via a network. The number of computers executing the computer programs is not particularly limited. For example, multiple computers (e.g., multiple servers) may execute the computer programs in cooperation with each other.

[0170] <7. Supplementary Information> While preferred embodiments of the present disclosure have been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear to any person with ordinary skill in the art of the present disclosure that various modifications or alterations may be conceived within the scope of the technical ideas described in the claims, and these will naturally also fall within the technical scope of the present disclosure.

[0171] For example, in the above embodiment, user U1 flies the mobile body 20 in advance and causes the imaging device 206 mounted on the mobile body 20 to capture images for generating a virtual object, but this technology is not limited to such examples. For example, if a virtual object has been generated by some method, such as when a virtual object has been generated in the past based on images captured using the mobile body 20 by a user other than user U1, the already generated virtual object may be used.

[0172] Furthermore, although the above embodiment was described assuming that the mobile body 20 is a drone, the mobile body 20 may be any mobile device. For example, the technology of this disclosure can be applied to various types of flying vehicles, such as drones. Moreover, the technology of this disclosure can be applied to manipulators, such as the hand or arm of a robot. In this case, the information processing device may, for example, control the display of a virtual object that the manipulator is handling on a display screen. Furthermore, the information processing device can generate movement information to control the movement of a mobile body, such as the fingertip of the manipulator. This makes it possible to generate movement information to control the movement of the fingertip of the manipulator in a more intuitive manner.

[0173] Furthermore, in the above embodiment, information regarding virtual objects and waypoints is recorded in the information processing device 100. However, information regarding virtual objects and waypoints may also be recorded in various servers connected to the network. In this case, the information processing device 100 can receive information recorded in the servers via the network as appropriate and generate movement information, imaging information, etc.

[0174] Furthermore, in the above embodiment, the user terminal 10 was described assuming that it was mainly a smartphone or tablet terminal. However, it is not limited to these, and the user terminal 10 may be a general-purpose PC (Personal Computer), a game console, a robot, or a wearable device such as an HMD (Head Mounted Display) or smartwatch.

[0175] Furthermore, the steps shown in the flowchart of the above embodiment include not only processes that are performed chronologically in the order described, but also processes that are not necessarily performed chronologically, but can be performed in parallel or individually. It goes without saying that even for steps that are performed chronologically, the order can be changed as appropriate in some cases.

[0176] Furthermore, the effects described herein are merely descriptive or illustrative and not limiting. In other words, the technology relating to this disclosure may produce other effects that will be apparent to those skilled in the art from the description herein, in addition to or in lieu of the effects described herein.

[0177] Furthermore, the following configurations also fall within the technical scope of this disclosure. (1) A display control unit that controls the display of virtual objects based on real-world objects on a display screen, It comprises a movement information generation unit that generates movement information for controlling the movement of a moving object, Information processing device. (2) The movement information generation unit generates the movement information based on the operation performed by the user viewing the display screen. The information processing device described in (1) above. (3) The aforementioned display includes the path of the moving object. The information processing device described in (2) above. (4) At least a portion of the aforementioned path displays one or more adjustment sections for adjusting the path. The operation described above is the operation of moving the position of the adjustment part displayed on the display screen. The information processing device described in (3) above. (5) The virtual object is displayed superimposed on the image captured by the first imaging device. An information processing device as described in any one of items (2) to (4) above. (6) The operation includes moving the viewpoint of the first imaging device, The movement information generation unit generates the movement information based on the movement of a predetermined position relative to the viewpoint. The information processing device described in (5) above. (7) The movement information generation unit generates the movement information based on an operation to move a designated object that specifies the path of the moving object. An information processing device as described in any one of items (2) to (6) above. (8) The moving body is equipped with a second imaging device for capturing images of the landscape. The system further includes an imaging information generation unit that generates imaging information for controlling the imaging range of the second imaging device based on user operations. An information processing device as described in any one of items (1) to (7) above. (9) The imaging information generation unit generates directional information relating to the direction in which the second imaging device takes images as imaging information. The information processing device described in (8) above. (10) The display screen shows the image captured by the first imaging device. The imaging information generation unit generates the orientation information based on an operation to move the orientation of the first imaging device. The information processing device described in (9) above. (11) The imaging information generation unit generates imaging information for controlling the field of view of the second imaging device based on a pinch-out or pinch-in operation performed by the user on the display screen. An information processing device as described in any one of items (8) to (10) above. (12) The system further includes an imaging prediction unit that predicts the image to be captured by the second imaging device based on the aforementioned movement information and imaging information. An information processing device as described in any one of items (8) to (11) above. (13) The system further includes a movement prediction unit that predicts the movement of the moving object based on the aforementioned movement information. An information processing device as described in any one of the above items (1) to (12). (14) The aforementioned moving body is capable of moving in three dimensions. An information processing device as described in any one of the above items (1) to (13). (15) The aforementioned moving object is an flying object. The information processing device described in (14) above. (16) The processor, Controlling the display of virtual objects based on real-world objects on the display screen, This includes generating movement information for controlling the movement of a moving object, Information processing methods. (17) On the computer, A function that controls the display of virtual objects based on real-world objects on the display screen, A function to generate movement information for controlling the movement of a moving object, A program to achieve this. [Explanation of Symbols]

[0178] 10 User terminals 100 Information Processing Devices 110 Imaging Unit 120 Sensor section 130 Input section 140 Acquisition Department 150 Processing Unit 151 Detection unit 157 Movement information generation unit 158 Imaging Information Generation Unit 159 Display information generation section 161 Movement prediction unit 162 Imaging prediction unit 170 Display Control Unit 175 Display section 20 Mobile Units 202 aircraft 204 Propeller 206 Imaging device Route 402 404 Virtual Route 406, 408, 410, 412 Waypoint 422, 432 virtual objects 610 display screen 612 Images of virtual objects Image of 614 virtual paths Image of 616 Waypoint 622 Designated object

Claims

1. The imaging device mounted on the mobile vehicle captures images of objects existing in real space. The processor generates scale information based on the scale of the captured real space, A display control step in which the processor displays, from the viewpoint of the imaging device, on a display screen, based on the generated scale information, an object existing in real space captured by the imaging device and a virtual path which is the path the moving object travels, while the moving object is moving. Having, Information processing methods.

2. The processor generates a virtual path, which is the path a moving object travels, based on the user's actions on the display screen, in a virtual path generation step that matches the scale of the real space. The processor includes a display control step of displaying an object existing in real space and the generated virtual path on the display screen, The virtual path is displayed superimposed on the image captured by the imaging device mounted on the moving body. The operation includes an operation to move the viewpoint of the imaging device. Information processing methods.

3. A movement information generation step in which the processor generates movement information adjusted based on the operation of the user viewing the display screen from the displayed virtual path, Having, The information processing method according to claim 1 or 2.

4. A plane detection step in which the processor performs plane detection based on information captured by the imaging device, Having, The information processing method according to claim 1 or 2.

5. At least a portion of the virtual path contains one or more adjustment parts for adjusting the virtual path. The following will be displayed: The operation described above is the operation of moving the position of the adjustment part displayed on the display screen. The information processing method according to claim 3.

6. The virtual path is displayed superimposed on the image captured by the imaging device. The information processing method according to claim 1.

7. In the virtual path generation step, the virtual path is generated based on an operation to move the designated object that specifies the path of the moving object. The information processing method according to claim 2.

8. The moving body is capable of moving in three dimensions based on the generated movement information. The information processing method according to claim 3.

9. The aforementioned moving object is an flying object. The information processing method according to any one of claims 1 to 8.

10. An information processing system comprising a mobile device and an information processing device, The aforementioned moving body is It is equipped with an imaging device that photographs objects existing in real space, The aforementioned information processing device is A generation unit that generates scale information based on the scale of the captured real space, A display control unit displays, from the viewpoint of the imaging device, an object existing in real space captured by the imaging device and a virtual path which is the path the moving object travels, on a display screen based on the generated scale information, while the moving object is moving. Equipped with, Information processing system.

11. Computers, An imaging device mounted on a moving object, which photographs an object existing in real space, and a generation unit that generates scale information based on the scale of the real space captured by the imaging device, A display control unit displays, based on the generated scale information, an object existing in real space captured by the imaging device and a virtual path which is the path the moving object travels, on a display screen from the viewpoint of the imaging device while the moving object is moving. A program designed to function as such.

Images