Image processing system, program, and image processing method

Abstract

To provide an image processing system, a program, and an image processing method for generating video with entertainment property while primarily handling captured video.SOLUTION: In a video processing system, a video processing device 6 executes the steps of: acquiring celestial body data obtained by a sensor mounted on a flying body 2; acquiring position of the flying body, data acquisition field angle, and acquisition posture; generating a celestial body image using the celestial body data; generating a virtual object image; generating a three-dimensional celestial body model containing a surface layer of the celestial body's surface and an object layer where the virtual object image is placed; placing the virtual object image; generating an object layer image on which the virtual object image is placed; and composing the celestial body image and the object layer image to generate a composite image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, a program, and an image processing method, and more particularly to processing video images captured by a camera mounted on an aircraft flying above a celestial body. [Background technology]

[0002] Conventionally, a method for displaying photographed images has been proposed in which the position of a photographed image of the Earth's surface photographed from the air is identified in three dimensions, the photographed range of the photographed Earth's surface is calculated and determined, the photographed image is transformed to fit the photographed range, and then the photographed image is displayed superimposed on a two-dimensional map (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2004 / 113836 Summary of the Invention [Problem to be solved by the invention]

[0004] The photographed video display method described in Patent Document 1 aims to check the consistency between the video information and the map, and pastes the photographed video onto a two-dimensional map. Therefore, the main purpose is not to view the photographed video taken from the sky, and the video generated by the photographed video display method of Patent Document 1 lacks entertainment value.

[0005] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing device, a program, and an image processing method that are capable of generating entertaining images while still using primarily photographed images. [Means for solving the problem]

[0006] One aspect of the present invention is an image processing device that includes a data acquisition unit that acquires data on celestial bodies obtained by a sensor mounted on an aircraft flying above the celestial body; a data information acquisition unit that acquires the position of the aircraft relative to the celestial body, and the acquired field of view and attitude of the sensor relative to the celestial body; an image generation unit that generates an image of the celestial body using the data on the celestial body acquired from the sensor; a virtual object generation unit that generates a virtual object image of a virtual object corresponding to the position of the aircraft, the acquired field of view and attitude of the sensor; and a composite image generation unit that combines the image of the celestial body and the virtual object image to generate a composite image.

[0007] One aspect of the present invention is a program that causes a computer to function as: data acquisition means for acquiring data on celestial bodies obtained by a sensor mounted on an aircraft flying above the celestial body; data information acquisition means for acquiring the position of the aircraft relative to the celestial body, the angle of view of the sensor from which the data relative to the celestial body was acquired, and the acquired attitude; image generation means for generating an image of the celestial body using the data on the celestial body acquired from the sensor; virtual object generation means for generating a virtual object image of a virtual object corresponding to the position of the aircraft, the angle of view of the sensor, and the acquired attitude; and composite image generation means for generating a composite image by combining the celestial body image and the virtual object image.

[0008] One aspect of the present invention is an image processing method that acquires data on a celestial body obtained by a sensor mounted on an aircraft flying above the celestial body, acquires the position of the aircraft relative to the celestial body, and the sensor's acquired angle of view and acquired attitude relative to the celestial body, generates an image of the celestial body using the data on the celestial body acquired from the sensor, generates a virtual object image of a virtual object corresponding to the position of the aircraft, the sensor's acquired angle of view and acquired attitude, and synthesizes the celestial body image and the virtual object image to generate a composite image. [Effects of the Invention]

[0009] According to the present invention, it is possible to generate entertaining video images while still using primarily photographed video images. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram of a video processing system according to the first embodiment. [Figure 2] FIG. 2 is a block diagram of the video processing device 6. [Figure 3] Figure 3 is a conceptual diagram of a three-dimensional Earth model. [Figure 4] FIG. 4 is a diagram for explaining the ground layer and the object layer. [Figure 5] FIG. 5 is a diagram showing an example in which an object image is placed at a position on the object layer of a three-dimensional earth model that corresponds to the object's position on the earth. [Figure 6] FIG. 6 is a diagram showing an example of the imaging range on a three-dimensional Earth model that is captured by the camera 11 of the space station 10. [Figure 7] FIG. 7 is a block diagram of a video processing device 6 configured as a computer system. [Figure 8] FIG. 8 is an example of an image captured by the camera 11 mounted on the space station 10 at time t. [Figure 9] FIG. 9 is a diagram showing an example of a virtual image of an object layer in the shooting range on a three-dimensional earth model under the conditions of orbital data at time t. [Figure 10] FIG. 10 is a diagram showing an example of a superimposed image at time t in which a virtual image at time t is superimposed on an image (real image) of the Earth photographed at time t. [Figure 11] FIG. 11 is an example of the superimposed image displayed on the terminal 8. [Figure 12] FIG. 12 is a diagram showing an example in which the object image is the predicted trajectory of the space station 10 and the estimated time of arrival of the space station 10 on the predicted trajectory. [Figure 13] FIG. 13 is a diagram showing an example in which the object image is a national flag. [Figure 14] FIG. 14 is a schematic diagram of a video processing system according to the second embodiment. [Figure 15]FIG. 15 is a diagram showing an example in which the object image is an SNS message. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] First Embodiment 1 is an overview diagram of a video processing system according to the first embodiment. In the following explanation, the Earth will be used as an example of a celestial body, but the celestial body is not limited to the Earth and may be other planets such as Mars, satellites such as the Moon, or fixed stars.

[0013] In Figure 1, 1 is the Earth, 2 is the flying vehicle, 3 is the parabolic antenna, 4 is the orbital data server, 5 is the object data server, 6 is the video processing device, 7 is the network, and 8 is the terminal.

[0014] The aircraft 2 is an aircraft that flies above the Earth. Examples of the aircraft 2 include a satellite, a spaceship or space station, an airplane, a helicopter, a drone, etc. In this embodiment, a space station 10 that orbits the Earth 1 while maintaining a fixed orbit will be used as an example. The space station 10 is equipped with a camera 11 with a predetermined angle of view, and the camera 11 can capture images of the surface of the Earth 1.

[0015] The parabolic antenna 3 receives images captured by the camera 11 from the space station 10 .

[0016] The orbit data server 4 transmits to the video processing device 6 orbit data including the shooting position of the space station 10 (time, latitude, longitude, altitude), the shooting angle of the camera 11 and the shooting attitude (attitude of the space station 10).

[0017] The object data server 5 transmits object data including an image of the object and the object's position on Earth 1 to the video processing device 6. The object may be of any type, including text information indicating a geographical location (such as a country name or city name), a national flag, a landmark, the flight path of the aircraft 2 (the orbit of the space station 10), or even a message on an SNS message sent from a specific geographical location or containing information about a location. The object image is an image (including text) of such an object. The object position may be of any type as long as it is information that identifies the object's position on Earth, including, for example, latitude, longitude, and altitude on Earth. Alternatively, the object may be a three-dimensional or two-dimensional coordinate value with a certain point or center of Earth (a celestial body) as the origin.

[0018] The video processing device 6 is a device that receives captured video from the space station 10, orbit data from the orbit data server 4, and object data from the object data server 5, and generates a superimposed video in which an object image located at a position on the Earth corresponding to the object position is superimposed on the captured video. The configuration of the video processing device 6 will be described in detail later.

[0019] The network 7 refers to a communication path that allows data communication. That is, the network 7 includes a LAN such as a dedicated line (dedicated cable) for direct connection or Ethernet (registered trademark), as well as a communication network such as a telephone communication network, a cable network, or the Internet, regardless of whether the communication method is wired or wireless.

[0020] Terminal 8 is a terminal that receives the superimposed video and displays the superimposed video. It can connect to network 7 via a wireless communication base station or the like to perform data communication. Terminal 8 is, for example, a smartphone, a mobile phone, a personal computer, a tablet computer, or the like. Basically, there are multiple terminals 8, each operated by a different user.

[0021] Next, a detailed description will be given of the configuration of the video processing device 6. FIG.

[0022] The video processing device 6 includes a video acquisition unit 21, a shooting information acquisition unit 22, an object data acquisition unit 23, a 3D modeling data storage device 24, a 3D earth model generation unit 25, an object placement unit 26, a virtual video generation unit 27, a video superimposition unit 28, and a video distribution unit 29.

[0023] The video acquisition unit 21 acquires video images taken by a camera 11 mounted on a space station 10 flying above the Earth 1.

[0024] The photography information acquisition unit 22 acquires orbit data from the orbit data server 4, including the photography position of the space station 10 (time, latitude, longitude, altitude), the photography angle of the camera 11, and the photography attitude (attitude of the space station 10).

[0025] The object data acquisition unit 23 acquires, from the object data server 5, object data including an image of the object and the object's position on the Earth.

[0026] The three-dimensional modeling data storage device 24 stores three-dimensional modeling data including three-dimensional data of the surface shape of the Earth 1, latitude and longitude data, and rotation and revolution information. Note that three-dimensional or two-dimensional coordinate values ​​with a certain point on the Earth (celestial body) as the origin may be used instead of the three-dimensional data or latitude and longitude data.

[0027] The 3D earth model generation unit 25 generates a 3D earth model of the earth as shown in Fig. 3 from the 3D modeling data storage device 24. As shown in Fig. 4, this 3D earth model has a ground surface layer of the earth's surface 1 and an object layer on which objects, which will be described later, are arranged.

[0028] The object placement unit 26 uses the object data acquired by the object data acquisition unit 23 to place the object image at a position on the object layer of the three-dimensional earth model that corresponds to the object's position on the earth 1. Fig. 5 is a diagram showing an example in which the object image is placed at a position on the object layer of the three-dimensional earth model that corresponds to the object's position on the earth. In the example of Fig. 5, objects AA to FF are placed on the object layer of the three-dimensional earth model.

[0029] The virtual image generation unit 27 estimates the imaging range on the 3D earth model captured by the camera 11 of the space station 10 under the conditions of the imaging position (time, latitude, longitude, altitude) of the space station 10, the imaging angle of view of the camera 11, and the imaging attitude of the camera 11 in the orbit data acquired by the imaging information acquisition unit 22. Figure 6 is a diagram showing an example of the imaging range on the 3D earth model captured by the camera 11 of the space station 10. Then, the virtual image generation unit 27 generates a virtual image of the object layer of the 3D earth model within that imaging range.

[0030] The image superimposition unit 28 superimposes the virtual image generated by the virtual image generation unit 27 onto the image (actual image) of the Earth 1 captured by the camera 11 of the space station 10 and acquired by the image acquisition unit 21, thereby generating a superimposed image.

[0031] The video distribution unit 29 distributes the superimposed video generated by the video superimposition unit 28 to the terminal 8 via the network 7. Note that the video distribution unit 29 does not necessarily have to be provided in the video processing device 6, and may be provided in another distribution device.

[0032] Specifically, the above-described video processing device 6 can be realized by a computer system having a processor that performs various types of arithmetic processing, etc. Fig. 7 is a block diagram of the video processing device 6 configured by a computer system.

[0033] As shown in Figure 7, the video processing device 6 can be configured by a computer 100 having a processor 101, memory (ROM or RAM) 102, input device (keyboard, mouse, touch panel, etc.) 103, communication device 104, and storage device (hard disk, semiconductor disk, etc.) 105.

[0034] In the video processing device 6, a program stored in the storage device 105 is loaded into the memory 102 and executed by the processor 101, thereby realizing video acquisition processing 111, shooting information acquisition processing 112, object data acquisition processing 113, 3D earth model generation processing 114, object placement processing 115, virtual video generation processing 116, video overlay processing 117, and video distribution processing 118. Here, the video acquisition processing 111 corresponds to the video acquisition unit 21, the shooting information acquisition processing 112 corresponds to the shooting information acquisition unit 22, the object data acquisition processing 113 corresponds to the object data acquisition unit 23, the 3D earth model generation processing 114 corresponds to the 3D earth model generation unit 25, the object placement processing 115 corresponds to the object placement unit 26, the virtual video generation processing 116 corresponds to the virtual video generation unit 27, the video overlay processing 117 corresponds to the video overlay unit 28, and the video distribution processing 118 corresponds to the video distribution unit 29. Furthermore, the three-dimensional modeling data storage device 24 corresponds to the storage device 105. Note that the storage device 105 (three-dimensional modeling data storage device 24) may be provided physically outside the computer 100 and connected to the computer 100 via a network such as a LAN.

[0035] Next, the operation of this embodiment will be described.

[0036] First, the object data acquisition unit 23 acquires, from the object data server 5, object data including an image of the object and the object position on the earth 1.

[0037] The object placement unit 26 uses the object data acquired by the object data acquisition unit 23 to pre-position the object image at a position on the object layer of the three-dimensional earth model that corresponds to the object position on the earth 1, as shown in Figure 5.

[0038] The image acquisition unit 21 acquires an image captured by the camera 11 mounted on the space station 10 at a certain shooting position (latitude, longitude, altitude), shooting angle of view of the camera 11, and shooting attitude (attitude of the space station 10 if the camera 11 is fixed) at a certain time t. Figure 8 is an example of an image captured by the camera 11 mounted on the space station 10 at time t.

[0039] The object data acquisition unit 23 acquires orbital data of the shooting position (latitude, longitude, altitude) at time t, the shooting angle of view of the camera 11, and the shooting attitude (the attitude of the space station 10 if the camera 11 is fixed), and outputs it to the virtual image generation unit 27.

[0040] The virtual image generation unit 27 estimates the imaging range on the 3D earth model captured by the camera 11 of the space station 10 under the conditions of the imaging position (time, latitude, longitude, altitude) of the space station 10, the imaging angle of view of the camera 11, and the imaging attitude of the camera 11 in the orbit data at time t acquired by the imaging information acquisition unit 22. The virtual image generation unit 27 then generates a virtual image of the object layer of the 3D earth model within that imaging range. Figure 9 is a diagram showing an example of a virtual image of the object layer of the imaging range on the 3D earth model under the conditions of the orbit data at time t.

[0041] The image superimposing unit 28 generates a superimposed image by superimposing the virtual image at time t generated by the virtual image generating unit 27 on the image (real image) of the Earth 1 at time t captured by the camera 11 of the space station 10 and acquired by the image acquiring unit 21. Fig. 10 is a diagram showing an example of the superimposed image at time t obtained by superimposing the virtual image at time t on the image (real image) of the Earth captured at time t.

[0042] The video distribution unit 29 distributes the superimposed video generated by the video superimposition unit 28 to the terminal 8 via the network 7 in a streaming manner.

[0043] Terminal 8 displays the superimposed video distributed from video distribution unit 29. FIG.

[0044] In the above description, examples of conceptual objects have been explained, but concrete examples of objects will be shown below.

[0045] 12 is a diagram showing an example in which the object image is the predicted trajectory of the space station 10 and the estimated time of arrival of the space station 10 on the predicted trajectory. In FIG. 12, the predicted trajectory and estimated time of arrival of the space station 10 are superimposed on the video captured by the space station 10.

[0046] 13 is a diagram showing an example in which the object image is a national flag. In Fig. 13, images of the national flags of various countries placed at the geographical locations of the countries are superimposed on the video captured by the space station 10.

[0047] In this embodiment, objects related to the subject of the footage captured by the space station (aircraft) are superimposed on the position of the subject, making it possible to generate entertaining footage while still focusing on the captured footage.

[0048] <Second embodiment> FIG. 14 is a schematic diagram of a video processing system according to the second embodiment.

[0049] In the second embodiment, the object data server 5 is provided with a function of collecting messages and message position information related to the messages as objects from a social network service (SNS) 9. Then, the object data server 5 converts the collected messages into object images and converts the message position information into object positions.

[0050] The message may contain information relating to the respective geographic location as well as information relating to the space station 10. Furthermore, the message location information may not only be the location from which the message was sent, but also a geographic location related to the content of the message.

[0051] The processing after conversion into the object image and object position is the same as in the first embodiment, and therefore the explanation will be omitted.

[0052] Fig. 15 is a diagram showing an example in which the object image is an SNS message. In Fig. 15, the SNS message is superimposed on the video captured by the space station 10, and the position of the SNS message corresponds to the geographical location where the SNS message was sent.

[0053] In the second embodiment, SNS messages are collected and superimposed on the captured video at a relevant position (for example, the position where the message was sent), so that video representation with real-time characteristics can be achieved.

[0054] Although the present invention has been described above by way of preferred embodiments, the present invention is not necessarily limited to the above-described embodiments, and can be modified and implemented in various ways within the scope of its technical concept. [Explanation of symbols]

[0055] 1 earth 2. Aircraft 3 Parabolic antenna 4 Orbital Data Server 5 Object Data Server 6. Video Processing Device 7 Network 8 Terminals 9. Social Media 21 Video acquisition unit 22 Shooting information acquisition unit 23 Object data acquisition unit 24 3D modeling data storage device 25 3D Earth Model Generation Unit 26 Object placement section 27 Virtual image generation unit 28 Video superimposition unit 29 Video Distribution Department 100 computers 101 processors 102 memory 103 Storage device 104 Input Device 105 Communication equipment

Claims

1. a data acquisition unit that acquires data of a celestial body obtained by a sensor mounted on an aircraft flying above the celestial body; a data information acquisition unit that acquires the position of the aircraft relative to the celestial body, the data acquisition angle of the sensor relative to the celestial body, and the acquired attitude; an image generating unit that generates an image of a celestial body using the data of the celestial body acquired from the sensor; a virtual object generation unit that generates a virtual object image that is an image of a virtual object that does not exist in the celestial body image; a three-dimensional celestial body model generation unit that generates a three-dimensional celestial body model of the celestial body, the three-dimensional celestial body model including a surface layer of the shape of the celestial body surface and an object layer on which the virtual object image is placed; an object placement unit that places the virtual object image at a position on the object layer of the three-dimensional celestial body model that corresponds to a geographical position of the virtual object on the celestial body; a virtual image generation unit that generates an object layer image when the sensor captures an image of the object layer on which the virtual object image is arranged, based on the shooting position of the aircraft, the data acquisition angle of the sensor relative to the celestial body, and the acquired attitude; a composite image generation unit that combines the celestial body image and the object layer image to generate a composite image; An image processing device comprising:

2. the composite image generation unit combines an astronomical body image generated from data acquired from the astronomical body at time t with an object layer image obtained by photographing the object layer on which the virtual object image is arranged, the object layer corresponding to the acquisition range of the data at time t, to generate a composite video over time. The image processing device according to claim 1 .

3. The air vehicle is a drone, a spacecraft, a satellite, or a space station.

3. The image processing device according to claim 1.

4. The celestial body is a planet, a satellite, or a star, including the Earth.

4. The image processing device according to claim 1.

5. Computer, a data acquisition means for acquiring data on a celestial body obtained by a sensor mounted on a flying vehicle flying above the celestial body; a data information acquisition means for acquiring the position of the aircraft relative to the celestial body, the data acquisition angle of the sensor relative to the celestial body, and the acquisition attitude of the sensor; an image generating means for generating an image of a celestial body using the data of the celestial body acquired from the sensor; a virtual object generating means for generating a virtual object image, which is an image of a virtual object that does not exist in the celestial body image; a three-dimensional celestial body model generating means for generating a three-dimensional celestial body model of the celestial body, the three-dimensional celestial body model including a surface layer of the shape of the celestial body surface and an object layer on which the virtual object image is placed; an object placement means for placing the virtual object image at a position on the object layer of the three-dimensional celestial body model corresponding to a geographical position of the virtual object on the celestial body; a virtual image generation means for generating an object layer image when the sensor captures an image of the object layer on which the virtual object image is arranged, based on the photographing position of the flying vehicle, the data acquisition angle of the sensor with respect to the celestial body, and the acquired attitude; a composite image generating means for generating a composite image by combining the celestial body image and the object layer image; A program that functions as a

6. acquiring data on the celestial body obtained by a sensor mounted on an aircraft flying above the celestial body; Acquire the position of the aircraft relative to the celestial body, the data acquisition angle of the sensor relative to the celestial body, and the acquisition attitude; generating an image of the celestial body using the data of the celestial body acquired from the sensor; generating a virtual object image of a virtual object that is an image of a virtual object that does not exist in the celestial body image; generating a three-dimensional celestial model of the celestial body, the three-dimensional celestial model including a surface layer of the shape of the celestial body surface and an object layer on which the virtual object image is placed; placing the virtual object image at a position of the object layer of the three-dimensional celestial body model corresponding to a geographical position of the virtual object on the celestial body; generating an object layer image when the sensor captures an image of the object layer on which the virtual object image is arranged, based on the shooting position of the flying vehicle, the data acquisition angle of the sensor with respect to the celestial body, and the acquired attitude; synthesizing the celestial body image and the object layer image to generate a composite image; Image processing methods.

Images