Aerial image display system and aerial image display program

Abstract

To provide a system for displaying aerial images that enables confirmation of the situation from any viewpoint.SOLUTION: An aerial image display system 100 includes a server 10 that stores information about a three-dimensional model MD, a camera device 20 that captures aerial images AP while flying, a display device 30 that displays a superimposed image SV together with visual confirmation of the camera device 20, and a computer 40. The computer 40 includes an information acquisition unit 41 that acquires at least information about the three-dimensional model MD and information about the aerial images AP, an image generation unit 42 that generates a superimposed image SV on the basis of the acquired information, and an information transmission unit 43 that transmits at least the generated superimposed image SV to the display device 30.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a system for displaying aerial images and a program executed by a computer for displaying aerial images. [Background technology]

[0002] Conventionally, there is known a technique for superimposing a 3D model on a captured image of a real space. For example, in Patent Document 1, a camera, a sensor, etc. provided on a device are used to grasp the surrounding environment and the current location of the device, and a superimposed image in which a 3D model is superimposed on the captured image is displayed on the monitor of the device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-156279 Summary of the Invention

[0004] Although the technology of Patent Document 1 described above can obtain superimposed images, it is necessary to check the situation from a ground viewpoint via a device such as a tablet. This can result in the burden of preparation and travel time required to carry the device and move it to the location where it will be used. Furthermore, it is difficult to ensure safety when using the device in a high place without footholds or in an area that is difficult to access. For these reasons, a technology that enables checking the situation from any viewpoint, not limited to a ground viewpoint, is desirable.

[0005] In view of the above, an object of the present invention is to provide a system for displaying aerial footage that enables checking the situation from any viewpoint, and a program to be executed by a computer for displaying aerial footage. [Means for solving the problem]

[0006] The technical means of the present invention for solving this technical problem is characterized as follows: The aerial image display system of the present invention comprises a server that stores information about a 3D model displayed in a space of a 3D coordinate system, a flyable camera that captures aerial images while in flight, a display device that is configured to allow the actual flight state of the camera device to be visible and that displays a superimposed image in which the 3D model is superimposed on the aerial image while the camera device is visible, and a computer that is capable of communicating information with each of the server, the camera device, and the display device. The computer comprises at least an information acquisition unit that acquires information about the 3D model and information about the aerial image, an image generation unit that generates the superimposed image based on the acquired information, and an information transmission unit that transmits at least the generated superimposed image to the display device.

[0007] In the aerial image display system of the present invention, the information acquisition unit further acquires information indicating the flight state of the imaging device, and the image generation unit generates the superimposed image by applying one or more of the information indicating the flight state of the imaging device, including position information of the imaging device, the direction of travel of the imaging device, the degree of inclination of the imaging device, the camera direction of the imaging device, and the camera angle of the imaging device, to the three-dimensional coordinate system of the three-dimensional model.

[0008] In the aerial image display system of the present invention, the information transmission unit further transmits to the display device at least one or more of the flight speed of the imaging device, the flight altitude of the imaging device, and the remaining battery charge of the imaging device as information indicating the flight status of the imaging device, so that the information is displayed on the display device together with the superimposed image.

[0009] In the aerial image display system of the present invention, the information transmission unit transmits the generated superimposed image to a terminal other than the display device, in addition to transmitting information to the display device.

[0010] In the aerial image display system of the present invention, the display device, when worn by the operator of the imaging device, is capable of covering the operator's eyes while ensuring a transparent field of view, and is equipped with a display unit that displays the transmitted superimposed image so that it can be seen by the operator.

[0011] The aerial image display program of the present invention causes a computer to execute an information acquisition means for acquiring at least information on a three-dimensional model displayed in a three-dimensional coordinate system space and information on aerial image captured in flight using a flying camera, an image generation means for generating a superimposed image in which the three-dimensional model is superimposed on the aerial image based on the acquired information, and an information transmission means for transmitting at least the generated superimposed image to a display device that is configured to allow the actual flight state of the camera device to be viewed and that displays the superimposed image together with the view of the camera device. [Effects of the Invention]

[0012] According to the present invention, it is possible to check the situation from any viewpoint. This reduces preparation and travel time and improves work efficiency by displaying a 3D model from any viewpoint. Furthermore, safety can be ensured even when used at high places without scaffolding or in areas that are difficult to access. In particular, even before construction, by overlaying a 3D model, the situation at completion can be confirmed by aerial photography. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is an overall schematic diagram of an aerial image display system according to an embodiment of the present invention; [Figure 2] 2 is a diagram showing an example of creating a three-dimensional model using the model creation terminal shown in FIG. 1. FIG. [Figure 3] 2 is a diagram showing a state in which the imaging device shown in FIG. 1 is flying near the planned construction site and capturing aerial images. FIG. [Figure 4]2 is a diagram showing an example of a case where a three-dimensional model, an aerial image, and flight information are acquired by the computer shown in FIG. 1, and a superimposed image is generated. FIG. [Figure 5] FIG. 2 is a diagram showing an example of a case in which the display device shown in FIG. 1 displays superimposed images and flight information while the operator visually confirms, via the display unit, how the camera device is actually flying and capturing aerial footage. [Figure 6] 1 is a flowchart illustrating a process for displaying superimposed images on a display device and a terminal using an aerial image display system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] 1, an aerial image display system 100 according to an embodiment of the present invention is a system for displaying an image obtained by superimposing a predetermined 3D model on an aerial image. The aerial image display system 100 includes a server 10, an image capturing device 20, a display device 30, a computer 40, and a terminal 50.

[0016] The server 10 is, for example, a cloud server or the like, and includes an information storage unit 11. The information storage unit 11 may be, for example, a non-volatile memory or the like, and stores information about the 3D model. The server 10 includes a communication device (not shown) and is capable of communicating information with the computer 40.

[0017] 2, the three-dimensional model MD is a model displayed in the space of a three-dimensional coordinate system. The three-dimensional model MD may be created in advance, for example, by a model creation terminal 60 separate from the computer 40, before aerial photography is performed by the image capture device 20. A predetermined application may be started in the model creation terminal 60, and after creating the three-dimensional model MD, the three-dimensional model MD may be transmitted to the server 10.

[0018] The three-dimensional model MD may be, for example, a replica of the shape of a building to be constructed on a planned construction site, etc. As shown in Fig. 3, when a planned construction site CS is specified, the three-dimensional model MD is created based on the area and shape of the planned construction site CS, the design values ​​of the building, etc.

[0019] As shown in FIG. 2, in this embodiment, the three-dimensional model MD is modeled after a rectangular parallelepiped building to be erected on the planned construction site CS. The three-dimensional model MD has a rectangular parallelepiped shape, with one vertex at the bottom corresponding to the origin of a three-dimensional coordinate system. In the three-dimensional model MD, the vertical, horizontal, and height directions are aligned with the X-axis, Y-axis, and Z-axis of the three-dimensional coordinate system. Note that the shape of the three-dimensional model MD is not limited to a rectangular parallelepiped shape, and may be any shape that can be displayed in the space of the three-dimensional coordinate system.

[0020] 1, the image capturing device 20 is capable of flying and is provided with a camera 21 in front of it. The image capturing device 20 captures aerial images while flying via the camera 21. The image capturing device 20 may be an unmanned aerial vehicle (e.g., a UAV, a drone, etc.) operated by an operator.

[0021] The photographing device 20 has a communication device (not shown) and is capable of communicating information with the computer 40. More specifically, for example, an operator's transmitter sends operation instructions and control signals to the photographing device 20 via wireless communication. Information photographed and detected by the photographing device 20 may also be sent to the computer 40 via the transmitter.

[0022] 3, for example, when the image capturing device 20 flies near the planned construction site CS, the camera 21 is directed from the sky toward the planned construction site CS on the ground. The camera 21 of the image capturing device 20 captures an aerial image AP as a bird's-eye view including the planned construction site CS. This aerial image AP is transmitted from the image capturing device 20 to the computer 40.

[0023] The photographing device 20 also includes a position detection device (for example, RTK-GNSS, etc.), various sensors (for example, an acceleration sensor, a battery sensor, etc.), etc. This allows the photographing device 20 to detect information indicating the flight state of the photographing device 20 (hereinafter, sometimes referred to as "flight information").

[0024] Examples of the "flight information" include the position information of the camera 20, the direction of travel of the camera 20, the degree of inclination of the camera 20, the direction of the camera 21 of the camera 20, the angle of the camera 21 of the camera 20, the flight speed of the camera 20, the flight altitude of the camera 20, and the remaining battery charge of the camera 20. This flight information is transmitted from the camera 20 to the computer 40.

[0025] 1, the computer 40 includes an information acquisition unit 41, an image generation unit 42, and an information transmission unit 43. The information acquisition unit 41, the image generation unit 42, and the information transmission unit 43 are each composed of an electric / electronic circuit, a program stored in a CPU, etc. The computer 40 has a communication device (not shown), and is capable of communicating information with the server 10, the image capture device 20, and the display device 30.

[0026] The information acquisition unit 41 acquires at least information on the three-dimensional model MD and information on the aerial footage AP. The information on the three-dimensional model MD and the information on the aerial footage AP are acquired from the server 10 and the image capture device 20, respectively. The information acquisition unit 41 further acquires information indicating the flight status of the image capture device 20.

[0027] 4, the image generating unit 42 generates a superimposed image based on the information acquired by the information acquiring unit 41. For example, the image generating unit 42 of the computer 40 superimposes a three-dimensional model MD on the aerial image AP, and executes a process of generating the superimposed image SV.

[0028] In the process of generating the superimposed image SV by the image generation unit 42, the flight information, such as the position information of the camera device 20, the traveling direction of the camera device 20, the tilt degree of the camera device 20, the direction of the camera 21 of the camera device 20, and the angle of the camera 21 of the camera device 20, is adapted to the three-dimensional coordinate system of the three-dimensional model MD. When adapting the three-dimensional coordinate system, one or more of the above-mentioned information may be used as the flight information. In this embodiment, the origin of the coordinate system of the rectangular parallelepiped three-dimensional model MD may be aligned with a corner of the rectangular planned construction site CS so that the bottom of the three-dimensional model MD overlaps with the planned construction site CS.

[0029] The information transmitting unit 43 transmits at least the generated superimposed image SV to the display device 30 and the terminal 50. The information acquiring unit 43 further transmits information indicating the flight state of the image capturing device 20 to the display device 30 so that the information is displayed on the display device 30 together with the superimposed image SV.

[0030] 5, for example, the information transmitting unit 43 transmits flight information FI to the display device 30 so that the flight information FI is displayed on the display unit 31 of the display device 30 together with the display of the superimposed image SV. The flight information FI displayed on the display unit 31 is the flight speed of the image capturing device 20, the flight altitude of the image capturing device 20, and the remaining battery charge of the image capturing device 20. When displayed on the display unit 31, one or more of the above-mentioned information may be used as the flight information.

[0031] As shown in Figures 1 and 5, the display device 30 is configured to enable visual confirmation of the actual flight state of the image capturing device 20, and displays the superimposed image SV along with visual confirmation of the image capturing device 20. For this purpose, the display device 30 is provided with a display unit 31. When worn by the operator of the image capturing device 20, the display unit 31 can ensure a see-through field of view while covering the operator's eyes. The display unit 31 is configured to display the transmitted superimposed image SV so that it can be visually confirmed by the operator.

[0032] More specifically, the display device 30 (display unit 31) may be a head-up display that can be worn on the operator's head. The display unit 31 is preferably made of a material that transmits and reflects visible light. When the operator operates the image capturing device 20, the operator may be able to view the actual flight state of the image capturing device 20 through the display unit 31, and the superimposed image SV may be visible as it is reflected on the display unit 31.

[0033] 5, for example, the actual flight state of the image capturing device 20 may be visible in its entirety when displayed on the display unit 31. Alternatively, the superimposed image SV and flight information FI may be displayed separately in a portion of the display unit 31. Note that the mode of the display device 30 (display unit 31) is not limited to the above and may be any mode as long as the operator can view the actual flight state of the image capturing device 20 and can view the superimposed image SV and flight information FI.

[0034] 1, the information transmitting unit 43 may transmit the generated superimposed video SV to a terminal 50 separate from the display device 30, in addition to transmitting information to the display device 30. The information transmission to the terminal 50 is executed, for example, from the computer 40. The terminal 50 is an external terminal separate from the computer 40, and includes a video display unit 51.

[0035] The video display unit 51 displays the transmitted superimposed video SV so that it can be viewed. The terminal 50 may be installed at a distance from the construction site CS, the image capturing device 20, and the operator. This allows the superimposed video SV to be viewed remotely, separately from the display device 30.

[0036] The actual operation of the aerial image display system 100 will be described with reference to the flowchart shown in Fig. 6. When the superimposed image SV is displayed on the display device 30 and the terminal 50, first, in step S1 of Fig. 6, a three-dimensional model MD is created by the model creation terminal 60, and the created three-dimensional model is stored in the server 10 (see Fig. 2).

[0037] Next, in step S2, an aerial image AP of the planned construction site CS is captured by the camera device 20 (see FIG. 3). Next, in step S3, the three-dimensional model MD, the aerial image AP, and flight information are acquired by the computer 40 (information acquisition unit 41). The information on the three-dimensional model MD is acquired from the server 10, and the aerial image AP and flight information are acquired from the camera device 20.

[0038] Next, in step S4, the computer 40 (image generation unit 42) generates a superimposed image SV (see FIG. 4). In generating the superimposed image SV, flight information, such as the position information of the image capture device 20, the direction of travel of the image capture device 20, the degree of inclination of the image capture device 20, the direction of the camera 21 of the image capture device 20, and the angle of the camera 21 of the image capture device 20, is adapted to the three-dimensional coordinate system of the three-dimensional model MD, and the three-dimensional model MD is superimposed on the aerial image AP.

[0039] Next, in step S5, the superimposed image SV and flight information are transmitted by the computer 40 (information transmission unit 43). The superimposed image SV is transmitted to the display device 30 and the terminal 50. The flight information is transmitted to the display device 30.

[0040] Next, in step S6, the display device 30 (display unit 31) displays flight information FI together with the superimposed image SV (see FIG. 5). The flight information FI displayed on the display unit 31 is the flight speed of the camera device 20, the flight altitude of the camera device 20, and the remaining battery charge of the camera device 20. When displayed on the display device 30, the superimposed image SV and flight information FI are displayed while the actual flight status of the camera device 20 is visually confirmed by the operator. The superimposed image SV is also displayed on the terminal 50.

[0041] [Effects of the embodiment] As described above, the aerial image display system 100 according to an embodiment of the present invention includes a server 10 that stores information about a three-dimensional model MD displayed in a three-dimensional coordinate system space, a flying camera 20 that captures aerial image AP in a flying state, a display device 30 that is configured to allow the actual flying state of the camera 20 to be viewed and that displays a superimposed image SV in which the 3D model MD is superimposed on the aerial image AP while the camera 20 is viewed, and a computer 40 that can communicate information with each of the server 10, the camera 20, and the display device 30. The computer 40 includes an information acquisition unit 41 that acquires at least information about the 3D model MD and the aerial image AP, an image generation unit 42 that generates the superimposed image SV based on the acquired information, and an information transmission unit 43 that transmits at least the generated superimposed image SV to the display device 30.

[0042] According to this, the display device 30 displays a superimposed image SV in which the 3D model MD is superimposed on the aerial image AP while the camera device 20 is visible, making it possible to check the situation from any viewpoint. This reduces preparation and travel time and improves work efficiency in displaying the 3D model from any viewpoint. Furthermore, safety can be ensured even when used in high places without scaffolding or in areas that are difficult to access. In particular, even before construction, by superimposing the 3D model, the situation at completion can be checked from the aerial image.

[0043] The information acquisition unit 41 further acquires information indicating the flight state of the camera device 20, and the image generation unit 42 generates the superimposed image SV by applying one or more of the information indicating the flight state of the camera device 20, including the position information of the camera device 20, the direction of travel of the camera device 20, the degree of inclination of the camera device 20, the camera direction of the camera device 20, and the camera angle of the camera device 20, to the three-dimensional coordinate system of the three-dimensional model MD.

[0044] This allows the three-dimensional model MD to appropriately follow the flight of the image capturing device 20. Therefore, it is possible to use the highly accurate superimposed image SV to more appropriately check the situation from a free viewpoint.

[0045] The information transmitting unit 43 further transmits, to the display device 30, at least one or more of the flight speed of the photographing device 20, the flight altitude of the photographing device 20, and the remaining battery charge of the photographing device 20 as information indicating the flight status of the photographing device 20, so that the information is displayed on the display device 30 together with the display of the superimposed image SV.

[0046] This allows flight information of the image capturing device 20 to be displayed on the display device 30. Therefore, the information indicating the flight state can be confirmed while visually checking the actual flight state of the image capturing device 20.

[0047] In addition to transmitting information to the display device 30, the information transmitting unit 43 transmits the generated superimposed image SV to a terminal 50 separate from the display device 30. The terminal 50 is installed away from the image capturing device 20 and the operator, and the superimposed image SV can be remotely confirmed separately from the display device 30.

[0048] The display device 30, when worn by an operator of the imaging device 20, is capable of covering the operator's eyes while ensuring a transparent field of view, and is equipped with a display unit 31 on which the transmitted superimposed image SV is displayed so as to be visible to the operator.

[0049] This allows the actual flying state of the image capturing device 20 and the superimposed image SV to be visually recognized hands-free and reliably, and therefore allows the operation of flying the image capturing device 20 to be performed appropriately. [Explanation of symbols]

[0050] 10...server, 20...imaging device, 21...camera, 30...display device, 31...display unit, 40...computer, 31...information storage unit, 41...information acquisition unit, 42...image generation unit, 43...information transmission unit, 50...terminal, 100...aerial image display system 100, AP...aerial image, FI...flight information, MD...3D model, SV...superimposed image

Claims

1. a server that stores information about a three-dimensional model displayed in a space of a three-dimensional coordinate system; A flying imaging device that captures aerial footage while flying; a display device configured to allow the actual flight state of the image capturing device to be visually recognized, and to display a superimposed image in which the three-dimensional model is superimposed on the aerial image while the image capturing device is visually recognized; a computer capable of communicating information with each of the server, the imaging device, and the display device; Equipped with The computer an information acquisition unit that acquires at least information on the three-dimensional model and information on the aerial image; an image generating unit that generates the superimposed image based on the acquired information; an information transmitting unit that transmits at least the generated superimposed image to the display device; Equipped with Aerial video display system.

2. 2. The aerial image display system according to claim 1, The information acquisition unit Furthermore, information indicating the flight state of the imaging device is acquired, The image generation unit The superimposed image is generated by applying, to a three-dimensional coordinate system of the three-dimensional model, one or more of the following information indicating the flight state of the imaging device: position information of the imaging device; a traveling direction of the imaging device; a tilt degree of the imaging device; a camera direction of the imaging device; and a camera angle of the imaging device. Aerial video display system.

3. 3. The aerial image display system according to claim 2, The information transmission unit Furthermore, as information indicating the flight status of the photographing device, at least one or more of the flight speed of the photographing device, the flight altitude of the photographing device, and the remaining battery charge of the photographing device are transmitted to the display device so as to be displayed on the display device together with the superimposed image. Aerial video display system.

4. The aerial image display system according to any one of claims 1 to 3, The information transmission unit In addition to transmitting information to the display device, the generated superimposed image is transmitted to a terminal other than the display device. Aerial video display system.

5. The aerial image display system according to any one of claims 1 to 3, The display device includes: a display unit that, when worn by an operator of the imaging device, can ensure a through-field of view while covering the eyes of the operator, and that displays the transmitted superimposed image so as to be visible to the operator; Equipped with Aerial video display system.

6. 5. The aerial image display system according to claim 4, The display device includes: a display unit that, when worn by an operator of the imaging device, can ensure a through-field of view while covering the eyes of the operator, and that displays the transmitted superimposed image so as to be visible to the operator; Equipped with Aerial video display system.

7. On the computer, an information acquisition means for acquiring at least information on a three-dimensional model displayed in a space of a three-dimensional coordinate system and information on aerial images captured in a flying state using a flying imaging device; an image generating means for generating a superimposed image in which the three-dimensional model is superimposed on the aerial image based on the acquired information; an information transmitting means for transmitting at least the generated superimposed image to a display device configured to allow visual confirmation of an actual flight state of the image capturing device and displaying the superimposed image together with visual confirmation of the image capturing device; Run Aerial image display program.

Images