Image processing apparatus, program, and image processing method

Abstract

To generate entertaining images while primarily using filmed images.SOLUTION: An image processing apparatus includes a data acquisition unit that acquires data on celestial object obtained by a sensor mounted on a flight vehicle flying over the celestial objects, a data information acquisition unit that acquires a position of the flight vehicle with respect to the celestial object, and an acquired angular field and an acquired posture with respect to the celestial object of the sensor, an image generation unit that generates a celestial image by using data on the celestial object acquired from the sensor, a virtual object generation unit that generates a virtual object image of a virtual object corresponding to the position of the flight vehicle, and the acquired angular field and the acquired posture of the sensor, and a composite image generation unit that synthesizes the celestial object image and the virtual object image to generate a composite image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing device, a program, and an image processing method, and more particularly to processing video images captured by a camera mounted on an aircraft flying above a celestial body. [Background technology]

[0002] Conventionally, a method for displaying photographed images has been proposed in which the position of a photographed image of the Earth's surface photographed from the air is identified in three dimensions, the photographed range of the photographed Earth's surface is calculated and determined, the photographed image is transformed to fit the photographed range, and then the photographed image is displayed superimposed on a two-dimensional map (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2004 / 113836 Summary of the Invention [Problem to be solved by the invention]

[0004] The photographed video display method described in Patent Document 1 aims to check the consistency between the video information and the map, and pastes the photographed video onto a two-dimensional map. Therefore, the main purpose is not to view the photographed video taken from the sky, and the video generated by the photographed video display method of Patent Document 1 lacks entertainment value.

[0005] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing device, a program, and an image processing method that are capable of generating entertaining images while mainly using photographed images. [Means for solving the problem]

[0006] One aspect of the present invention is an image processing apparatus including a data acquisition unit that acquires data of a celestial body obtained by a sensor mounted on a flying object flying above the celestial body, a data information acquisition unit that acquires the position of the flying object with respect to the celestial body, the acquisition angular range and acquisition attitude of the data of the sensor with respect to the celestial body, an image generation unit that generates a celestial body image using the data of the celestial body acquired from the sensor, a virtual object generation unit that generates a virtual object image of a virtual object corresponding to the position of the flying object, the acquisition angular range and acquisition attitude of the sensor, and a composite image generation unit that composites the celestial body image and the virtual object image to generate a composite image.

[0007] One aspect of the present invention is a program that causes a computer to function as a data acquisition means for acquiring data of a celestial body obtained by a sensor mounted on a flying object flying above the celestial body, a data information acquisition means for acquiring the position of the flying object with respect to the celestial body, the acquisition angular range and acquisition attitude of the data of the sensor with respect to the celestial body, an image generation means for generating a celestial body image using the data of the celestial body acquired from the sensor, a virtual object generation means for generating a virtual object image of a virtual object corresponding to the position of the flying object, the acquisition angular range and acquisition attitude of the sensor, and a composite image generation means for compositing the celestial body image and the virtual object image to generate a composite image.

[0008] One aspect of the present invention is an image processing method for acquiring data of a celestial body obtained by a sensor mounted on a flying object flying above the celestial body, acquiring the position of the flying object with respect to the celestial body, the acquisition angular range and acquisition attitude of the data of the sensor with respect to the celestial body, generating a celestial body image using the data of the celestial body acquired from the sensor, generating a virtual object image of a virtual object corresponding to the position of the flying object, the acquisition angular range and acquisition attitude of the sensor, and compositing the celestial body image and the virtual object image to generate a composite image.

Advantages of the Invention

[0009] According to the present invention, it is possible to generate an entertaining video while mainly using a captured video.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Embodiment for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0012] <First Embodiment> FIG. 1 is an overview diagram of a video processing system according to the first embodiment. In the following description, as an example of a celestial body, the Earth will be described as an example, but it is not limited to the Earth, and other planets such as Mars, satellites such as the Moon, and stars may also be used.

[0013] In FIG. 1, 1 is the Earth, 2 is a flying object, 3 is a parabolic antenna, 4 is an orbit data server, 5 is an object data server, 6 is a video processing device, 7 is a network, and 8 is a terminal.

[0014] The flying object 2 is a flying object flying over the Earth. Examples of the flying object 2 include various types such as satellites, spacecraft or space stations, airplanes, helicopters, and drones. In this embodiment, the space station 10 orbiting the Earth 1 while maintaining a certain orbit will be described as an example. The space station 10 is equipped with a camera 11 having a predetermined viewing angle, and the camera 11 can capture a ground image of the Earth 1.

[0015] The parabolic antenna 3 receives the captured video captured by the camera 11 from the space station 10.

[0016] The orbit data server 4 transmits orbit data including the shooting position (time, latitude and longitude, altitude) of the space station 10, the shooting angle of view of the camera 11, and the shooting posture (space station 10 posture) to the video processing device 6.

[0017] The object data server 5 transmits object data including the image of an object and the object position on the Earth 1 of the object to the video processing device 6. Here, the object can be of any type, for example, text of information indicating a geographical location (country name, city name, etc.), a national flag, a landmark, the flight path of the aircraft 2 (the orbit of the space station 10), or even an SNS message transmitted at a certain geographical location or including information regarding some position. And the object image is the image of these objects (including text). The object position can be of any type as long as it is information specifying the position of the object on the Earth, for example, latitude and longitude on the Earth, altitude, etc. Also, it can be three-dimensional or two-dimensional coordinate values with a certain point or the center of the Earth (celestial body) as the origin.

[0018] The video processing device 6 is a device that receives the captured video from the space station 10, the orbit data from the orbit data server 4, and the object data from the object data server 5, and generates a superimposed video in which the object image arranged at the position on the Earth corresponding to the object position is superimposed on the captured video. The details of the configuration of the video processing device 6 will be described later.

[0019] The network 7 means a communication path capable of data communication. That is, the network 7 includes a dedicated line (dedicated cable) for direct connection, a LAN such as Ethernet (registered trademark), as well as communication networks such as a telephone communication network, a cable network, and the Internet, and the communication method can be either wired or wireless.

[0020] The terminal 8 is a terminal that receives the superimposed video and displays the superimposed video. And it can be connected to the network 7 via a wireless communication base station or the like to perform data communication. The terminal 8 is, for example, a smartphone, a mobile phone, a personal computer, a tablet computer, etc. Basically, there are a plurality of terminals 8, which are operated by each user.

[0021] Next, the details of the configuration of the video processing device 6 will be described. FIG. 2 is a block diagram of the video processing device 6.

[0022] The image processing apparatus 6 includes an image acquisition unit 21, a shooting information acquisition unit 22, an object data acquisition unit 23, a three-dimensional modeling data storage device 24, a three-dimensional earth model generation unit 25, an object placement unit 26, a virtual image generation unit 27, an image superimposing unit 28, and an image distribution unit 29.

[0023] The image acquisition unit 21 acquires a captured image captured by a camera 11 mounted on a space station 10 flying over the sky of the earth 1.

[0024] The shooting information acquisition unit 22 acquires orbit data including the shooting position (time, latitude and longitude, altitude) of the space station 10, the shooting angle of view of the camera 11, and the shooting posture (the posture of the space station 10) from an orbit data server 4.

[0025] The object data acquisition unit 23 acquires object data including an image of an object and the object position on the earth of the object from an object data server 5.

[0026] The three-dimensional modeling data storage device 24 stores three-dimensional modeling data including three-dimensional data of the surface shape of the earth 1, latitude and longitude data, and rotation and revolution information. Note that instead of the three-dimensional data or the latitude and longitude data, three-dimensional or two-dimensional coordinate values with a certain point on the earth (celestial body) as the origin may be used.

[0027] The three-dimensional earth model generation unit 25 generates a three-dimensional earth model of the earth as shown in FIG. 3 from the three-dimensional modeling data storage device 24. As shown in FIG. 4, this three-dimensional earth model has a surface layer on the surface of the earth 1 and an object layer for placing an object to be described later.

[0028] The object placement unit 26 uses the object data acquired by the object data acquisition unit 23 to place the object image at the position of the object layer of the three-dimensional earth model corresponding to the object position on the earth 1 of the object. FIG. 5 is a diagram showing an example in which the object image is placed at the position of the object layer of the three-dimensional earth model corresponding to the object position on the earth of the object. In the example of FIG. 5, it is an example in which objects AA to FF are placed on the object layer of the three-dimensional earth model.

[0029] The virtual video generation unit 27 estimates the shooting range on the three-dimensional earth model photographed by the camera 11 of the space station 10 under the conditions of the shooting position (time, latitude and longitude, altitude) of the space station 10, the shooting angle of view of the camera 11, and the shooting posture of the camera 11, which are acquired by the shooting information acquisition unit 22. FIG. 6 is a diagram showing an example of the shooting range on the three-dimensional earth model photographed by the camera 11 of the space station 10. Then, the virtual video generation unit 27 generates a virtual video of the object layer of the three-dimensional earth model within the shooting range.

[0030] The video superimposing unit 28 superimposes the virtual video generated by the virtual video generation unit 27 on the photographed video (real video) of the earth 1 photographed by the camera 11 of the space station 10 acquired by the video acquisition unit 21 to generate a superimposed video.

[0031] The video distribution unit 29 distributes the superimposed video generated by the video superimposing unit 28 to the terminal 8 via the network 7. Note that the video distribution unit 29 does not necessarily need to be provided in the video processing apparatus 6, and may be provided in another distribution apparatus.

[0032] Specifically, the above-described video processing apparatus 6 can be realized by a computer system having a processor that performs various arithmetic processes and the like. FIG. 7 is a block diagram of the video processing apparatus 6 configured by a computer system.

[0033] As shown in FIG. 7, the video processing apparatus 6 can be configured by a computer 100 having a processor 101, a memory (ROM, RAM, etc.) 102, an input device (keyboard, mouse, touch panel, etc.) 103, a communication device 104, and a storage device (hard disk, semiconductor disk, etc.) 105.

[0034] In the video processing apparatus 6, when the program stored in the storage device 105 is loaded into the memory 102 and executed by the processor 101, a video acquisition process 111, a shooting information acquisition process 112, an object data acquisition process 113, a three-dimensional earth model generation process 114, an object placement process 115, a virtual video generation process 116, a video superimposition process 117, and a video distribution process 118 are realized. Here, the video acquisition process 111 corresponds to the video acquisition unit 21, the shooting information acquisition process 112 corresponds to the shooting information acquisition unit 22, the object data acquisition process 113 corresponds to the object data acquisition unit 23, the three-dimensional earth model generation process 114 corresponds to the three-dimensional earth model generation unit 25, the object placement process 115 corresponds to the object placement unit 26, the virtual video generation process 116 corresponds to the virtual video generation unit 27, the video superimposition process 117 corresponds to the video superimposition unit 28, and the video distribution process 118 corresponds to the video distribution unit 29. Also, the three-dimensional modeling data storage device 24 corresponds to the storage device 105. Note that the storage device 105 (three-dimensional modeling data storage device 24) may be physically provided outside the computer 100 and connected to the computer 100 via a network such as a LAN.

[0035] Next, the operation of this embodiment will be described.

[0036] First, the object data acquisition unit 23 acquires object data including an object image and an object position on the earth 1 of the object from the object data server 5.

[0037] The object placement unit 26 uses the object data acquired by the object data acquisition unit 23 to pre-place the object image at the position of the object layer of the three-dimensional earth model corresponding to the object position on the earth 1 of the object, as shown in Fig. 5.

[0038] The video acquisition unit 21 acquires the captured video taken by the camera 11 mounted on the space station 10 at a certain time t at the shooting position (latitude, longitude, altitude), the shooting angle of view of the camera 11, and the shooting posture (when the camera 11 is fixed, the posture of the space station 10). Fig. 8 is an example of the captured video taken by the camera 11 mounted on the space station 10 at time t.

[0039] The object data acquisition unit 23 acquires the orbital data of the shooting position (latitude, longitude, altitude), the shooting angle of view of the camera 11, and the shooting posture (when the camera 11 is fixed, the posture of the space station 10) at time t, and outputs it to the virtual video generation unit 27.

[0040] The virtual video generation unit 27 estimates the shooting range on the three-dimensional earth model captured by the camera 11 of the space station 10 under the conditions of the shooting position (time, latitude, longitude, altitude), the shooting angle of view of the camera 11, and the shooting posture of the orbital data at time t acquired by the shooting information acquisition unit 22. Then, the virtual video generation unit 27 generates a virtual video of the object layer of the three-dimensional earth model in the shooting range. Fig. 9 is a diagram showing an example of the virtual video of the object layer in the shooting range on the three-dimensional earth model under the conditions of the orbital data at time t.

[0041] The video superimposing unit 28 superimposes the virtual video at time t generated by the virtual video generation unit 27 on the captured video (real video) of the earth 1 taken by the camera 11 of the space station 10 acquired by the video acquisition unit 21 to generate a superimposed video. Fig. 10 is a diagram showing an example of the superimposed video at time t obtained by superimposing the virtual video at time t on the captured video (real video) of the earth at time t.

[0042] The video distribution unit 29 streams and distributes the superimposed video generated by the video superimposition unit 28 to the terminal 8 via the network 7.

[0043] The terminal 8 displays the superimposed video distributed from the video distribution unit 29. FIG. 11 shows an example of the superimposed video displayed on the terminal 8.

[0044] In the above description, examples of conceptual objects have been described, but examples of specific objects are shown below.

[0045] FIG. 12 is a diagram showing an example in which the object image is the predicted orbit of the space station 10 and the predicted arrival time of the space station 10 on the predicted orbit. In FIG. 12, the predicted orbit and the predicted arrival time of the space station 10 are superimposed on the captured video captured by the space station 10.

[0046] FIG. 13 is a diagram showing an example in which the object image is a national flag. In FIG. 13, images of national flags arranged at the geographical positions of each country are superimposed on the captured video captured by the space station 10.

[0047] In this embodiment, since the object related to the subject of the captured video captured by the space station (flying object) is superimposed on the position of the subject, it is possible to generate an entertaining video while mainly using the captured video.

[0048] <Second Embodiment> FIG. 14 is an overview diagram of a video processing system in the second embodiment.

[0049] In the second embodiment, the object data server 5 is provided with a function of collecting messages and message position information related to the messages as objects from the social network service (SNS) 9. Then, the object data server 5 converts the collected messages into object images and converts the message position information into object positions.

[0050] The message may contain information not only related to the space station 10 but also related to each geographical location. Further, the message position information may be not only the position where the message was transmitted but also the geographical location related to the content of the message.

[0051] Since the processing after conversion into the object image and the object position is the same as that in the first embodiment, the description thereof will be omitted.

[0052] FIG. 15 is a diagram showing an example in which the object image is an SNS message. In FIG. 15, an SNS message is superimposed on a captured video taken by the space station 10, and the position of the SNS message corresponds to the geographical location where the SNS message was transmitted.

[0053] In the second embodiment, since the SNS messages are collected and superimposed on the relevant positions (for example, the transmission positions of the messages) of the captured video, a real-time video expression can be performed.

[0054] Although the present invention has been described by way of preferred embodiments, the present invention is not necessarily limited to the above-described embodiments, and various modifications can be made and implemented within the scope of the technical idea thereof.

Description of Reference Numerals

[0055] 1 Earth 2 Aircraft 3 Parabolic Antenna 4 Orbit Data Server 5 Object Data Server 6 Video Processing Device 7 Network 8 Terminal 9 SNS 21 Video Acquisition Unit 22 Shooting Information Acquisition Unit 23 Object Data Acquisition Unit 24 3D Modeling Data Storage Device 25 3D Earth Model Generation Unit 26 Object Placement Unit 27 Virtual Image Generation Unit 28 Image Superimposition Unit 29 Image Distribution Unit 100 Computer 101 Processor 102 Memory 103 Storage Device 104 Input Device 105 Communication Device

Claims

1. A data acquisition unit that acquires data of the celestial body obtained by a sensor mounted on a flying object flying above the celestial body; A data information acquisition unit that acquires the position of the flying object with respect to the celestial body, the acquisition angle of view, and the acquisition posture of the data of the sensor with respect to the celestial body; An image generation unit that generates a celestial body image using the data of the celestial body acquired from the sensor; A virtual object generation unit that generates a virtual object image, which is an image of a virtual object that does not exist in the celestial body image; A three-dimensional celestial body model generation unit that generates a three-dimensional celestial body model of the celestial body, including a ground layer of the shape of the celestial body surface of the celestial body and an object layer on which the virtual object image is arranged; An object arrangement unit that arranges the virtual object image at the position of the object layer of the three-dimensional celestial body model corresponding to the geographical position of the virtual object on the celestial body; A virtual image generation unit that generates an object layer image when the sensor images the object layer on which the virtual object image is arranged, based on the shooting position of the flying object, the acquisition angle of view, and the acquisition posture of the data of the sensor with respect to the celestial body; A composite image generation unit that composites the celestial body image and the object layer image to generate a composite image; An image processing apparatus comprising the above.

2. The composite image generation unit composites a celestial body image generated from data acquired from the celestial body at time t and an object layer image when the object layer on which the virtual object image is arranged is imaged corresponding to the acquisition range of the data at time t, and generates a composite video over time. The image processing apparatus according to Claim 1.

3. The flying object is a drone, a spaceship, a satellite, or a space station. The image processing apparatus according to Claim 1 or Claim 2.

4. The celestial body is a planet, a satellite, or a star including the Earth. The image processing apparatus according to any one of Claims 1 to 3.

5. A computer, A data acquisition means for acquiring data of the celestial body obtained by a sensor mounted on a flying object flying above the celestial body; A data information acquisition means for acquiring the position of the flying object with respect to the celestial body, the acquisition angle of view, and the acquisition posture of the data of the sensor with respect to the celestial body; An image generation means for generating a celestial body image using the data of the celestial body acquired from the sensor; Virtual object generation means for generating a virtual object image, which is an image of a virtual object that does not exist in the celestial body image. 3D celestial body model generation means for generating a 3D celestial body model of the celestial body, including a ground layer of the shape of the celestial body surface of the celestial body and an object layer where the virtual object image is arranged. Object arrangement means for arranging the virtual object image at the position of the object layer of the 3D celestial body model corresponding to the geographical position of the virtual object on the celestial body. Virtual image generation means for generating an object layer image when the sensor images the object layer where the virtual object image is arranged, based on the shooting position of the flying object, the acquisition angular range and acquisition attitude of the data of the sensor with respect to the celestial body. Composite image generation means for combining the celestial body image and the object layer image to generate a composite image. A program that functions as such.

6. Obtaining data of the celestial body obtained by a sensor mounted on a flying object flying over the celestial body. Obtaining the position of the flying object with respect to the celestial body, the acquisition angular range and acquisition attitude of the data of the sensor with respect to the celestial body. Generating a celestial body image using the data of the celestial body obtained from the sensor. Generating a virtual object image of a virtual object, which is an image of a virtual object that does not exist in the celestial body image. Generating a 3D celestial body model of the celestial body, including a ground layer of the shape of the celestial body surface of the celestial body and an object layer where the virtual object image is arranged. Arranging the virtual object image at the position of the object layer of the 3D celestial body model corresponding to the geographical position of the virtual object on the celestial body. Generating an object layer image when the sensor images the object layer where the virtual object image is arranged, based on the shooting position of the flying object, the acquisition angular range and acquisition attitude of the data of the sensor with respect to the celestial body. Combining the celestial body image and the object layer image to generate a composite image. An image processing method.

Images