Information display system and information display method

Abstract

To provide an information display system and an information display method that are able to display an object in a virtual space superimposed on an image captured in a real space without requiring a large amount of training data.SOLUTION: An information display system includes: a drone equipped with a camera, a GPS module and an altitude sensor; a proportional controller; and a computer. An information computing unit of a computer comprises: an information acquisition unit for acquiring various types of information from the drone; and a display control unit for converting positional information, directional information and range information in a real space acquired by the information acquisition unit into positional information, directional information and range information in a virtual space, generating an image of a virtual object to be captured when the virtual space is shot with the camera, and displaying on a display the generated image of the virtual object superimposed on a real image. The display control unit makes an original point in the virtual space coincident with a reference position selected in the real space.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an information display system and an information display method. [Background technology]

[0002] Computers are now being used to design, construct, and manage buildings. Computer-designed buildings are generated as three-dimensional (3D) data on a computer.

[0003] If it were possible to overlay building data onto the scenery captured by a camera mounted on an unmanned aerial vehicle (UAV), including a drone, it would be possible to give users a virtual experience as if the building being designed were actually there.

[0004] However, while the 3D data of a building is expressed using a Cartesian coordinate system, the GPS module installed in a drone expresses its own position using a world coordinate system. Therefore, in order to overlay an image of a building designed on a computer onto the scenery captured by the camera, it is necessary to associate different coordinate systems. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2019-087229 A Summary of the Invention [Problem to be solved by the invention]

[0006] One of the methods for matching is the technology described in Patent Document 1, which estimates the position of a drone in a virtual space from an image. The technology described in Patent Document 1 estimates the position using a model trained by machine learning.

[0007] However, the technology described in Patent Document 1 uses machine learning, and therefore has a problem in that a large amount of learning data is required to determine a position in a virtual space.

[0008] In view of the above problems, the present invention aims to provide an information display system and an information display method that can superimpose objects in a virtual space onto an image captured in real space without requiring a large amount of learning data. [Means for solving the problem]

[0009] An information display system according to one aspect of the present invention comprises a flying object having a main body, a photographing device movably attached to the main body for photographing an image, and a position measuring device for measuring the position of the photographing device; an information acquisition unit that acquires from the flying object position information of the photographing device in real space defined by a world coordinate system, directional information of the photographing device relative to the main body, range information of an image photographed by the photographing device, and a real image photographed by the photographing device; and a display control unit that converts the position information, directional information, and range information in real space acquired by the information acquisition unit into position information, directional information, and range information of the photographing device in a virtual space to generate an image of a virtual object that would be photographed by the photographing device when photographing in the virtual space, and displays the image of the generated virtual object by superimposing it on the real image, and the display control unit matches a reference position selected in real space with an origin in the virtual space.

[0010] The virtual objects may be two or more buildings designed in a virtual space, and the display control unit may generate a virtual image for each of the buildings.

[0011] The reference position may be the position where the flying object starts flying.

[0012] The virtual space may be defined by a planar Cartesian coordinate system.

[0013] An information display system according to one aspect of the present invention includes an operation information calculation unit that calculates an identification code corresponding to the direction of movement or turning based on an operation of instructing a flying object in that direction, and a display control unit that may superimpose a figure corresponding to the direction of movement or turning on a real image based on the identification code calculated by the operation information calculation unit.

[0014] An information display method according to another aspect of the present invention includes an information acquisition step of acquiring, from a flying object having a main body, a photographing device movably attached to the main body for photographing an image, and a position measuring device for measuring the position of the photographing device, position information of the photographing device in a real space defined by a world coordinate system, directional information of the photographing device relative to the main body, range information of an image photographed by the photographing device, and a real image photographed by the photographing device; a display control step of converting the position information, directional information and range information in the real space acquired in the information acquisition step into position information, directional information and range information of the photographing device in a virtual space to generate an image of a virtual object that would be photographed when the photographing device photographs in the virtual space, and displaying the generated image of the virtual object by superimposing it on the real image; and a matching step of matching a reference position selected in the real space with an origin in the virtual space, wherein the matching step is executed before the information acquisition step is executed. Effect of the Invention

[0015] According to the information display system and information display method of the present invention, it is possible to superimpose an object in a virtual space onto an image captured in a real space and display it, without requiring a large amount of learning data. [Brief description of the drawings]

[0016] [Figure 1] 1 is a diagram showing an overview of an information display system according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating an example of a computer that realizes the functions of the information display system according to the first embodiment of the present invention. [Diagram 3]1 is a functional block diagram of an information display system according to a first embodiment of the present invention. [Figure 4] 2 is a diagram showing an example of a data structure used in the information display system according to the first embodiment of the present invention. FIG. [Diagram 5] 5 is a flowchart illustrating an example of the operation of the information display system according to the first embodiment of the present invention. [Figure 6] 6 is another flowchart illustrating an example of the operation of the information display system according to the first embodiment of the present invention. [Figure 7] FIG. 2 is a diagram illustrating the relationship between different coordinate systems in the information display system according to the first embodiment of the present invention. [Figure 8] 6 is another flowchart illustrating an example of the operation of the information display system according to the first embodiment of the present invention. [Figure 9] 2 is a diagram showing an example of an image displayed on a display included in the information display system in accordance with the first embodiment of the present invention. FIG. [Figure 10] FIG. 11 is a functional block diagram of an information display system according to a second embodiment of the present invention. [Figure 11] 13 is a diagram showing an example of a data structure used in an information display system according to a second embodiment of the present invention. FIG. [Figure 12] 13 is a diagram showing an example of an image displayed on a display included in an information display system in accordance with a second embodiment of the present invention. FIG. [Figure 13] FIG. 11 is a diagram showing an example of another image displayed on a display included in the information display system in accordance with the second embodiment of the present invention. [Figure 14] FIG. 11 is a diagram showing an example of another image displayed on a display included in the information display system in accordance with the second embodiment of the present invention. [Figure 15] FIG. 11 is a diagram showing an example of another image displayed on a display included in the information display system in accordance with the second embodiment of the present invention. [Figure 16]FIG. 11 is a diagram showing an example of another image displayed on a display included in the information display system in accordance with the second embodiment of the present invention. [Figure 17] FIG. 11 is a diagram showing an example of another image displayed on a display included in the information display system in accordance with the second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0018] (Information Display System 1) The information display system 1 according to the first embodiment shown in Fig. 1 includes a drone 2, a transmitter 3, and a computer 4. The flight of the drone 2 is controlled by the user operating the transmitter 3. The computer 4 stores 3D data of the building in advance. The computer 4 generates an image that makes the building, stored as 3D data, appear to be located on the planned construction site according to the position, direction, etc. of the drone 2 flying over the planned construction site.

[0019] (Drone 2) The drone 2 comprises a main body 21 equipped with a propeller, a control circuit, a battery, etc., a camera 22 attached to the main body 21, a communication device 23, a GPS (Global Positioning System) module 24, a direction sensor 25, a field of view information setting acquisition unit 26, and an altitude sensor 27.

[0020] The GPS module 24 is a device that receives radio waves emitted by multiple GPS satellites and compares them to output information related to the position of the GPS module 24. Since the GPS module 24 is disposed near the center of the main body 21 and close to the camera 22, hereinafter, the position of the GPS module 24 may be referred to as the position of the drone 2 or the position of the camera 22.

[0021] The direction sensor 25 is a device that measures information about the direction in which the main body 21 and the camera 22 are facing. The direction sensor 25 includes, for example, an electronic compass and a gyro sensor, and can measure the direction in which the main body 21 is facing and the relative direction in which the camera 22 is facing with respect to the direction in which the main body 21 is facing.

[0022] The angle-of-view information setting and acquiring unit 26 is a device that switches the zoom magnification, angle of view, etc. of the camera 22. The angle-of-view information setting and acquiring unit 26 is, for example, a register that holds values ​​for setting and acquiring the magnification of a digital zoom connected to an image sensor of the camera 22. Since the zoom magnification and the angle of view correspond to each other, the angle of view of the camera 22 can be switched depending on the magnification value set in the angle-of-view information setting and acquiring unit 26. In addition, by reading out the magnification value set in the angle-of-view information setting and acquiring unit 26, the angle of view set in the camera 22 can be acquired.

[0023] The altitude sensor 27 is a device that includes a pressure sensor, measures air pressure, and outputs altitude. The drone 2 is an example of a flying object.

[0024] (Proposal 3) The transmitter 3 is wirelessly connected to the drone 2 and is a device that controls the drone 2 by communicating with it to instruct the drone 2 to ascend, descend, turn right, turn left, and move forward, backward, and left and right.

[0025] The transmitter 3 includes a main body 31, a left stick 32 and a right stick 33 that are arranged on the main body 31 and can be pushed down by the pilot holding the main body 31 with his or her hands and using the thumbs on both sides, and a communication device 34 that is built into the main body 31 and communicates with the drone 2 and a computer 4, which will be described later. The left stick 32 is arranged on the left side of the main body 31, and the right stick 33 is arranged on the right side of the main body 31.

[0026] When the user operates the left stick 32 and the right stick 33, the communication device 34 emits signals to change the altitude and direction of the flying drone 2, to rotate the drone 2, and to change the orientation of the camera 22. The communication device 34 communicates with the communication device 23 of the drone 2, for example, in the 2.4 GHz band.

[0027] Furthermore, the communicator 34 acquires images captured by the camera 22 of the drone 2 via the communicator 23. The communicator 34 transmits the acquired images to the computer 4 described later. The communicator 34 acquires information output by the GPS module 24, direction sensor 25, angle of view information setting acquisition unit 26, and altitude sensor 27 of the drone 2 via the communicator 23. The communicator 34 transmits the acquired information to the computer 4 described later.

[0028] The signals emitted by the communication device 34 when the left stick 32 and the right stick 33 are tilted or pressed in each direction, and the rules by which the communication device 23 detects these signals and controls the flight of the drone 2, are not limited to those described above.

[0029] (Computer 4) The computer 4 includes an information calculation device 41, a display 42, and a communication device 43. The information calculation device 41, which will be described in detail later, is a device that processes various information acquired from the drone 2 and displays the processed results on the display 42. The communication device 43 is, for example, a wireless network interface, and is a device that can communicate with the communication device 34 of the transmitter 3.

[0030] (Hardware configuration of information processing device 41) 2 shows an example of a hardware configuration of the information processing device 41. The information processing device 41 includes a processor 41a, a main memory unit 41b, a communication unit 41c, an auxiliary memory unit 41d, and a bus 41e. The processor 41a executes a program stored in the main memory unit 41b or the auxiliary memory unit 41d to realize functions described below.

[0031] (Information calculation unit 5) 3 shows the functional configuration of the information display system 1. The information calculation unit 5 includes an information acquisition unit 51 that acquires various pieces of information from the drone 2, and a display control unit 52 that controls the information to be displayed on the display 42 of the computer 4.

[0032] The position information measurement unit 61, the direction information measurement unit 62, the angle of view information measurement unit 63, the image capture unit 64, and the altitude measurement unit 65 correspond to the GPS module 24, the direction sensor 25, the angle of view information setting acquisition unit 26, the camera 22, and the altitude sensor 27, respectively, shown in FIG. 1. The position information measurement unit 61 has a function of measuring the position of the drone 2. The direction information measurement unit 62 has a function of measuring the shooting direction of the camera 22 of the drone 2. The angle of view information measurement unit 63 has a function of measuring the angle of view of an image captured by the camera 22. The altitude measurement unit 65 has a function of measuring the altitude of the drone 2. The image capture unit 64 has a function of capturing an image.

[0033] (Information acquisition unit 51) The information acquisition unit 51 is connected to the position information measurement unit 61, the direction information measurement unit 62, the angle of view information measurement unit 63, and the image capture unit 64. The information acquired by the information acquisition unit 51 will be described below. The information acquired by the information acquisition unit 51 from the drone 2 is not limited to that described below, and different information or additional information may be acquired.

[0034] (Data Structure) The information on the position of the camera 22 acquired by the information acquisition unit 51 includes, for example, position data 1 and 2 shown in Fig. 4. More specifically, the position data 1 and 2 are numerical values ​​indicating latitude and longitude.

[0035] The information related to the direction of the camera 22 acquired by the information acquisition unit 51 includes, for example, direction data 1 to 6 shown in Fig. 4. More specifically, the direction data 1 to 3 are x, y, and z components when the front direction of the main body 21 of the drone 2 is expressed as a 3D vector. More specifically, the direction data 4 to 6 are x, y, and z components when the front direction of the camera 22 relative to the front direction of the main body 21 of the drone 2 is expressed as a 3D vector.

[0036] The information related to the angle of view of camera 22 acquired by information acquisition unit 51 includes, for example, angle of view data 1 and 2 shown in Fig. 4. More specifically, angle of view data 1 and 2 are numerical values ​​that respectively represent the horizontal and vertical spreads of an image captured by camera 22 in degrees.

[0037] The image of the camera 22 acquired by the information acquisition unit 51 includes, for example, pixel data 1 to N shown in Fig. 4. More specifically, the pixel data 1 to N is the luminance of each image captured by the camera 22 arranged from the start point to the end point of scanning the image.

[0038] (Display control unit 52) The display control unit 52 performs control to superimpose 3D data of a building on the image captured by the camera 22 of the flying drone 2 and display it in AR, based on the information acquired from the information acquisition unit 51. The control performed by the display control unit 52 will be described below.

[0039] As shown in FIG. 5, the AR display process performed by the display control unit 52 roughly includes two steps: matching coordinates (S100) and display (S200).

[0040] FIG. 6 shows the details of the coordinate matching step (FIG. 5: S100). First, the display control unit 52 determines a reference position and a reference direction in the real space (S101). For example, after the user turns on the power of the drone 2 and the drone 2 is in a state where it can communicate with the computer 4 via the transmitter 3, the user operates the input device of the computer 4 to select the position of the drone 2 to start flying. In addition, the user, for example, operates the transmitter 3 to change the orientation of the camera 22 to an arbitrary position, and then operates the input device of the computer 4 to select the direction of the camera of the drone 2 at the time of operation. The information acquisition unit 51 acquires the position selected by the user from the position information measurement unit 61. The information acquisition unit 51 acquires the direction selected by the user from the direction information measurement unit 62. The display control unit 52 determines the position and direction acquired by the information acquisition unit 51 as the reference position and reference direction, respectively.

[0041] The method by which the information acquisition unit 51 determines the reference position and the reference direction is not limited to the above. The information acquisition unit 51 may detect an event such as the drone 2 taking off or the drone 2 being wirelessly connected to the transmitter 3 without a user selection, and determine the reference position and the reference direction.

[0042] Next, the display control unit 52 obtains a transformation parameter TP that associates coordinates in the real space with coordinates in the virtual space (S102). The display control unit 52 acquires the determined reference position P (latitude and longitude) of the drone 2. In addition, the display control unit 52 acquires a 3D vector D (dx, dy, dz) indicating the direction of the camera 22 from the direction information measurement unit 62 via the information acquisition unit 51.

[0043] More specifically, as shown in FIG. 7, the display control unit 52 obtains a conversion parameter TP that corresponds the position, direction, and scale between the real space and the virtual space. The latitude and longitude of the reference position P in the real space correspond to the origin O (0,0,0) in the virtual space. A known conversion formula can be used for conversion from the world coordinate system to the plane Cartesian coordinate system. The coordinate axes may be translated so that both the x coordinate and the y coordinate become 0 after conversion. Furthermore, the display control unit 52 may translate the z axis so that the altitude measured by the altitude measuring unit 65 at the reference position becomes an altitude of 0 in the virtual space. The reference direction D in the real space corresponds to the y direction in the virtual space. A unit distance (e.g., 1 meter) in the real space corresponds to a unit distance (e.g., 1 meter) in the virtual space.

[0044] The coordinates of the virtual space that the display control unit 52 aligns with the reference position in the real space may include any offset. Similarly, the direction of the virtual space that the display control unit 52 aligns with the reference direction in the real space may include any offset.

[0045] Through the above process, the process of associating coordinates (S100) shown in Fig. 5 is completed. Note that the determination of the reference position P and the determination of the reference direction D may be performed one before the other, or may be performed simultaneously.

[0046] Returning to FIG. 4, the display control unit 52 performs AR display (S200) using the transformation parameters TP determined in the coordinate association (FIG. 6: S102).

[0047] Hereinafter, with reference to FIG. 8, details of the AR display in which a building as 3D data designed on the computer 4 is superimposed on an image captured by the drone 2 and displayed on the display 42 will be described.

[0048] First, the information acquiring unit 51 acquires the location information from the location information measuring unit 61 (S201). The location information is, for example, the location data 1 (value 36.0613) and the location data 2 (value 140.0516) shown in FIG.

[0049] The information acquiring unit 51 acquires directional information from the directional information measuring unit 62 (S202). The directional information is, for example, directional data 1 (value 0.7327), directional data 2 (value 0.8523), directional data 3 (value -0.4291), directional data 4 (value 0.9631), directional data 5 (value 0.0448), and directional data 6 (value -0.3737) shown in FIG.

[0050] The information acquisition unit 51 acquires range information from the image shooting unit 64 (S203). The range information indicates the horizontal angle of view and the vertical angle of view of the image acquired in the next step, and is, for example, the angle of view data 1 (value 83.2192) and the angle of view data 2 (value 46.3229) shown in FIG.

[0051] The information acquisition unit 51 acquires an image from the image shooting unit 64 (S204). The image is, for example, the values ​​of each pixel data shown in FIG. 4 (luminance value 32, luminance value 29, ..., luminance value 215). Note that the pixels do not have to be represented by the luminance value of each pixel, and may include luminance values ​​for the red, green, and blue channels, or may include color difference information. The image may be compressed and encoded information.

[0052] The display control unit 52 generates a background image of the building (S205). The display control unit 52 draws, in an image buffer, for example, an image acquired by the information acquisition unit 51 from the image shooting unit 64 as an image to be displayed in the background of the building.

[0053] The display control unit 52 draws the building on the background image (S206). The display control unit 52 generates an image of the building captured by the virtual camera VC from the transformation parameters TP obtained in step S102 and the position information, direction information, and range information obtained in steps S201 to S203. For example, the display control unit 52 converts the coordinates of the camera in the real space into a position in the virtual space by the transformation parameters TP. The display control unit 52 also converts the relative camera direction in the real space represented by the front direction of the drone 2 and the camera direction relative to the front direction into the absolute direction of the camera in the real space. Next, the display control unit 52 converts the camera direction in the real space into a direction in the virtual space by the transformation parameters TP. The size of the image of the building to be generated is obtained by assuming that there is a virtual screen in the range of the horizontal angle of view in the horizontal direction and the range of the vertical angle of view in the vertical direction, centered on the direction of the virtual camera VC. Then, the display control unit 52 renders the image of the building in an image buffer corresponding to the obtained camera direction and screen size. If there are two or more buildings, rendering is performed for each building.

[0054] By executing steps S201 to S206, an AR image in which building data is superimposed on a background image captured in real space can be displayed.

[0055] Incidentally, following step S206, the display control unit 52 may superimpose the position information in the real space, the transformation parameters TP, the position information in the virtual space after being transformed by the transformation parameters TP, and the like stored in the image buffer on the display.

[0056] (Image of AR display) FIG. 9 is an example of an image EI in which two buildings VB are superimposed on an image BG captured by the camera 22. Since the buildings VB are displayed on top of the image BG, a part of the image BG appears to be hidden by the buildings VB, and the image is recognized as a natural image. In the illustrated example, an area SD is superimposed, which indicates position information (latitude, longitude) in the real space and position information (x coordinate, y coordinate, z coordinate) in the virtual space after conversion by the conversion parameter TP, in letters and numbers. By superimposing the area SD on the image BG and the buildings VB, the user can obtain a clue when selecting a reference position and a reference direction, and can intuitively correct the correspondence between the coordinates in the real space and the coordinates in the virtual space.

[0057] As described above, the information display system 1 does not use machine learning and does not require a large amount of learning data.

[0058] Furthermore, since the reference position and reference direction selected by the user and determined by the display control unit 52 are associated with the origin O in the virtual space, the user of the information display system 1 does not need to perform on-site surveying, measurement of orientation, etc. Therefore, according to the information display system 1, no complicated operations are required to associate coordinates in the real space with those in the virtual space, and the correspondence can be performed intuitively.

[0059] In this manner, AR display for one frame is completed. This is repeated every time the camera 22 captures a new image or every time a time equivalent to a predetermined frame rate has elapsed.

[0060] The image captured by the camera 22 is displayed, making it appear as if the drone 2 is flying in a virtual space and that a building is actually present.

[0061] The GPS information is an example of location information, and is not limited to this. The information display system 1 may include a GNSS module that measures GNSS information other than GPS information, instead of the GPS module 24.

[0062] (Second embodiment) The information display system 1 is a system that displays objects in a virtual space, images captured in a real space, and various information acquired by an information acquisition unit 51. In contrast, the information display system 7, the functional configuration of which is shown in Fig. 10, is a system that displays information related to the operation of the drone 2, in addition to objects in a virtual space and images captured in a real space. Below, differences from the information display system 1 will be described with reference to Figs. 10, 11, and 1.

[0063] The information calculation unit 8 of the information display system 7 includes, in addition to the information acquisition unit 51 and the display control unit 52, an operation information calculation unit 81 that calculates information regarding the information for operating the drone 2 input from the transmitter 3 shown in Figure 1.

[0064] (Data Structure) The information on the operation of the drone 2 calculated by the operation information calculation unit 81 includes, for example, operation data 1 to 3 shown in Fig. 11. More specifically, the operation data 1 to 3 are an identification code (ID) indicating the direction of a rotation operation, an ID indicating the direction of a translation operation, and an ID indicating the direction of an ascent / descent operation, respectively.

[0065] The operation data 1 is data that is, for example, a value "0" when no parallel movement operation is performed in the front direction of the main body 21, and is one of the following values ​​"1" to "4" when a parallel movement operation is performed in the forward / backward or left / right direction. For example, when the left stick 32 of the transmitter 3 shown in FIG. 1 is tilted upward, the operation information calculation unit 81 calculates a value "1" as the operation data 1. When the left stick 32 is tilted downward, the operation information calculation unit 81 calculates a value "2" as the operation data 1. When the right stick 33 is tilted leftward, the operation information calculation unit 81 calculates a value "3" as the operation data 1. When the right stick 33 is tilted rightward, the operation information calculation unit 81 calculates a value "4" as the operation data 1.

[0066] As shown in Figures 12 to 15, when a parallel movement operation is performed in each of the forward / backward / left / right directions, the display control unit 52 displays arrows OA indicating each direction superimposed on the image BG and the building VB in accordance with the values ​​calculated by the operation information calculation unit 81 corresponding to each direction.

[0067] The operation data 2 is data that, for example, has a value of "0" when no turning operation is performed, a value of "1" when a left (counterclockwise) turning operation is performed, and a value of "2" when a right (clockwise) turning operation is performed. For example, when the left stick 32 is tilted leftward, the operation information calculation unit 81 calculates a value of "1" as the operation data 2. When the left stick 32 is tilted rightward, the operation information calculation unit 81 calculates a value of "2" as the operation data 2.

[0068] As shown in Figures 16 and 17, when a turning movement operation is performed in each of the left and right directions, the display control unit 52 displays arrows OA indicating each direction superimposed on the image BG and the building VB in accordance with the values ​​calculated by the operation information calculation unit 81 corresponding to each direction.

[0069] The operation data 3 is data that, for example, has a value of "0" when no up / down operation is being performed, a value of "1" when an up operation is being performed, and a value of "2" when a down operation is being performed. For example, when the right stick 33 is tilted upward, the operation information calculation unit 81 calculates a value of "1" as the operation data 3. When the right stick 33 is tilted downward, the operation information calculation unit 81 calculates a value of "2" as the operation data 3.

[0070] When an up or down operation is performed in either the up or down direction, the display control unit 52 displays the arrow OA shown in Figures 14 and 15 pointing upward or downward, respectively, superimposed on the image BG and the building VB, depending on the value calculated by the operation information calculation unit 81 corresponding to each direction.

[0071] When two or more of the operation data 1 to 3 have values ​​other than "0", the display control unit 52 may display multiple corresponding arrows overlapping each other, or may display another arrow that combines the multiple arrows. For example, when the pilot tilts both the left stick 32 and the right stick 33 upward to move the drone 2 forward while ascending, the operation information calculation unit 81 calculates the operation data 1 to 3 shown in FIG. 11. The display control unit 52 may display the arrow OA pointing to the back shown in FIG. 12 and the arrow OA pointing upward shown in FIG. 18 overlapping each other, or may display these arrows OA side by side, according to the operation data 1 to 3. In addition, the display control unit 52 may display an arrow OA with the tip of the arrow OA shown in FIG. 12 raised toward the front side of the screen, which indicates an operation that combines a forward translation operation and an ascending operation, for example.

[0072] By displaying a figure calculated based on information obtained from transmitter 3 in addition to information obtained from drone 2, it becomes easier for the pilot of drone 2 to control the drone, allowing him or her to more easily approach the part of the building that he or she wants to see.

[0073] The present invention allows various embodiments and modifications without departing from the broad spirit and scope of the present invention. The above-described embodiments are for the purpose of explaining the present invention and do not limit the scope of the present invention. That is, the scope of the present invention is indicated by the claims, not the embodiments. Various modifications made within the scope of the claims and the scope of the invention equivalent thereto are considered to be within the scope of the present invention. [Explanation of symbols]

[0074] 1, 7 Information display system 2. Drone 21 Main unit 22 Camera 23 Communication Device 24 GPS Module 25 Directional Sensor 26 Field of view information setting acquisition section 27 Altitude Sensor 3. Propo 31 Main unit 32 Left Stick 33 Right Stick 34 Communication Device 4. Computer 41 Information processing device 42 Display 43 Communication Device 5, 8 Information calculation section 51 Information Acquisition Department 52 Display control unit 61 Location information measurement section 62 Direction information measurement section 63 Angle of view information measurement section 64 Image capture unit 65 Altitude measurement section 81 Operation information calculation section

Claims

1. a flying object including a main body, an image capturing device movably attached to the main body and configured to capture an image, and a position measuring device configured to measure the position of the image capturing device; an information acquisition unit that acquires, from the flying object, position information of the image capturing device in a real space defined by a world coordinate system, direction information of the image capturing device relative to the main body, range information of an image captured by the image capturing device, and a real image captured by the image capturing device; a display control unit that converts the position information, the direction information, and the range information in the real space acquired by the information acquisition unit into position information, direction information, and range information of the image capture device in a virtual space, generates an image of a virtual object that will be captured by the image capture device in the virtual space, and displays the generated image of the virtual object by superimposing it on the real image, the display control unit causes the reference position selected in the real space to coincide with coordinates in the virtual space that include an arbitrary offset from an origin; Information display system.

2. the virtual objects are two or more buildings designed in the virtual space; the display control unit generates an image of the virtual object for each of the buildings. The information display system according to claim 1 .

3. The reference position is a position where the flying object starts flying. The information display system according to claim 1 .

4. The virtual space is defined by a plane orthogonal coordinate system. The information display system according to claim 1 .

5. an operation information calculation unit that calculates an identification code corresponding to a direction of movement or rotation based on an operation to instruct the flying object in the direction of movement or rotation; the display control unit displays a graphic corresponding to the direction of movement or rotation by superimposing it on the real image based on the identification code calculated by the operation information calculation unit. The information display system according to claim 1 .

6. an information acquisition step of acquiring, from a flying object including a main body, an image capturing device movably attached to the main body for capturing images, and a position measuring device for measuring the position of the image capturing device, position information of the image capturing device in a real space defined by a world coordinate system, direction information of the image capturing device relative to the main body, range information of an image captured by the image capturing device, and a real image captured by the image capturing device; a display control step of converting the position information, the direction information, and the range information in the real space acquired in the information acquisition step into position information, direction information, and range information of the image capturing device in a virtual space, generating an image of a virtual object that will be captured by the image capturing device in the virtual space, and displaying the generated image of the virtual object by superimposing it on the real image; a correspondence step of matching the reference position selected in the real space with coordinates in the virtual space including an arbitrary offset from an origin, The associating step is performed before the information obtaining step is performed. Information display method.

Images