IMAGE GENERATION DEVICE, IMAGE GENERATION METHOD, AND PROGRAM

Abstract

The present technology relates to an image generation device, an image generation method, and a program which enable an arbitrary point on the earth to be observed from a free viewpoint in the sky at an arbitrary time. The image generation device is provided with an image generation unit which uses a 3D model of a stationary subject, which is generated by using a satellite image captured by an artificial satellite, and dynamic subject specifying information specifying a dynamic subject, and generates a free viewpoint image obtained by imaging a prescribed point on the earth from a prescribed virtual viewpoint in the sky. The present technology can be applied to an image generation device which generates a free viewpoint image from a satellite image captured by the artificial satellite.

Description

[Technical field]

[0001] The present technology relates to an image generation device, an image generation method, and a program, and in particular to an image generation device, an image generation method, and a program that enable observation of any point on the ground at any time from any viewpoint in the sky. [Background technology]

[0002] Earth observation is carried out by observation satellites equipped with imaging devices (see, for example, Patent Document 1). Particularly in recent years, the number of small observation satellites in low orbit has been increasing.

[0003] In the technical field of VR (Virtual Reality), a 3D model is used that allows an object as a subject to be reproduced and displayed from any arbitrary viewpoint. The three-dimensional data of the 3D model of the subject is converted, for example, into a plurality of texture images and depth images captured from a plurality of viewpoints, transmitted to a reproduction device, and displayed on the reproduction side (for example, see Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2006-277007 A [Patent Document 2] International Publication No. 2017 / 082076 Summary of the Invention [Problem to be solved by the invention]

[0005] In observing the Earth, there is a demand to be able to observe any point on the ground from any viewpoint in the sky at any time, just like with VR. However, in the case of geostationary satellites, because they are located at an altitude of 36,000 km, it is difficult to observe with high resolution, and because the satellite's position is fixed, only specific points can be observed. In addition, in the case of satellites orbiting in low or medium orbit, the view can only be seen from a viewpoint in orbit, and not from a free viewpoint. Therefore, at present, the demand to observe any point on the ground from any viewpoint in the sky at any time is not being fully met.

[0006] This technology was developed in light of these circumstances, and makes it possible to observe any point on the ground, at any time, and from any viewpoint in the sky. [Means for solving the problem]

[0007] This technology No. 1 The image generating device on the side includes an image generating unit that generates a free viewpoint image of a predetermined point on the ground viewed from a predetermined virtual viewpoint in the sky using a 3D model of a still subject generated using a satellite image captured by an artificial satellite and dynamic subject identification information that identifies a dynamic subject. The image generating unit generates a base image of the 3D model of the still subject viewed from a predetermined virtual viewpoint in the sky as a background image, generates a real-time image viewed from the predetermined virtual viewpoint in the sky as a foreground image using real-time external information and known information about the dynamic subject as the dynamic subject identification information, and generates the free viewpoint image by synthesizing the base image and the real-time image. . An image generating device according to a second aspect of the present technology includes an image generating unit that generates a free viewpoint image of a predetermined point on the ground viewed from a predetermined virtual viewpoint in the sky using a 3D model of a still subject generated using a satellite image captured by a satellite and dynamic subject identification information that identifies a dynamic subject, the image generating unit generates a base image viewed from the predetermined virtual viewpoint in the sky as a background image of the 3D model of the still subject, and generates a real-time image viewed from the predetermined virtual viewpoint in the sky as a foreground image using a 3D model of a hypothetical dynamic subject as the dynamic subject identification information, and generates the free viewpoint image by synthesizing the base image and the real-time image, the image generating unit generates a 3D model of the hypothetical dynamic subject using a near real-time satellite image captured by a satellite at a time that differs by several hours from a time specified by a user, and generates the real-time image by estimating the 3D model of the dynamic subject at the time specified by the user from the 3D model of the hypothetical dynamic subject using external information. An image generating device according to a third aspect of the present technology includes an image generating unit that generates a free viewpoint image of a predetermined point on the ground viewed from a predetermined virtual viewpoint in the sky using a 3D model of a still subject generated using a satellite image captured by a satellite and dynamic subject identification information that identifies a dynamic subject, the image generating unit generates a base image of the 3D model of the still subject viewed from the predetermined virtual viewpoint in the sky as a background image, generates a real-time image viewed from the predetermined virtual viewpoint in the sky as a foreground image using a 3D model of a hypothetical dynamic subject as the dynamic subject identification information, and generates the free viewpoint image by synthesizing the base image and the real-time image, the image generating unit generates a 3D model of the hypothetical dynamic subject using a quasi-real-time satellite image captured by a satellite at a time that differs by several hours from a time specified by a user, generates a quasi-real-time image of the dynamic subject viewed from the predetermined virtual viewpoint in the sky using the 3D model of the hypothetical dynamic subject, and generates the real-time image from the quasi-real-time image of the dynamic subject using external information.

[0008] This technology No. 1 In the side image generating method, an image generating device generates a free viewpoint image of a predetermined point on the ground viewed from a predetermined virtual viewpoint in the sky using a 3D model of a still subject generated using a satellite image captured by an artificial satellite and dynamic subject identification information that identifies a dynamic subject. and generating a base image of the 3D model of the still subject viewed from a predetermined virtual viewpoint in the sky as a background image, generating a real-time image viewed from the predetermined virtual viewpoint in the sky as a foreground image using real-time external information and known information about the dynamic subject as the dynamic subject identification information, and generating the free viewpoint image by synthesizing the base image and the real-time image. .

[0009] This technology No. 1The program on the side causes the computer to function as an image generating unit that generates a free viewpoint image of a predetermined point on the ground from a predetermined virtual viewpoint in the sky, using a 3D model of a still subject generated using a satellite image captured by an artificial satellite and dynamic subject identification information that identifies a dynamic subject. The image generating unit generates a base image of the 3D model of the still subject viewed from a predetermined virtual viewpoint in the sky as a background image, generates a real-time image viewed from the predetermined virtual viewpoint in the sky as a foreground image using real-time external information and known information about the dynamic subject as the dynamic subject identification information, and generates the free viewpoint image by synthesizing the base image and the real-time image. This is for the purpose.

[0010] This technology 1st~3rd On the other hand, a free viewpoint image of a specified point on the ground viewed from a specified virtual viewpoint in the sky is generated using a 3D model of a still subject generated using a satellite image captured by an artificial satellite and dynamic subject identification information that identifies a dynamic subject. In the first aspect, a base image of the 3D model of the still subject viewed from a predetermined virtual viewpoint in the sky is generated as a background image, a real-time image viewed from the predetermined virtual viewpoint in the sky is generated as a foreground image using real-time external information and known information about the dynamic subject as the dynamic subject identification information, and the free viewpoint image is generated by combining the base image and the real-time image. In the second aspect, a base image of the 3D model of the still subject viewed from a predetermined virtual viewpoint in the sky is generated as a background image, a real-time image viewed from the predetermined virtual viewpoint in the sky is generated as a foreground image using a 3D model of a hypothetical dynamic subject as the dynamic subject identification information, and the free viewpoint image is generated by combining the base image and the real-time image, a 3D model of the hypothetical dynamic subject is generated using a near real-time satellite image captured by an artificial satellite at a time that is several hours different from the time specified by the user, and the real-time image is generated by estimating the 3D model of the dynamic subject at the time specified by the user from the 3D model of the hypothetical dynamic subject using external information. In a third aspect, a base image of the 3D model of the still subject viewed from a predetermined virtual viewpoint in the sky is generated as a background image, a real-time image viewed from the predetermined virtual viewpoint in the sky is generated as a foreground image using a 3D model of a hypothetical dynamic subject as the dynamic subject identification information, the free viewpoint image is generated by combining the base image and the real-time image, a 3D model of the hypothetical dynamic subject is generated using a quasi-real-time satellite image captured by a satellite at a time that differs by several hours from the time specified by the user, a quasi-real-time image of the dynamic subject is generated by viewing the 3D model of the hypothetical dynamic subject from the predetermined virtual viewpoint in the sky, and the real-time image is generated from the quasi-real-time image of the dynamic subject using external information.

[0011] The image generating device according to one aspect of the present technology can be realized by causing a computer to execute a program. The program to be executed by the computer can be provided by transmitting the program via a transmission medium or by recording the program on a recording medium.

[0012] The image generating device may be an independent device, or may be an internal block constituting a single device. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing an example of the configuration of a satellite image processing system according to an embodiment to which the present technology is applied. [Diagram 2] FIG. 2 is a block diagram showing a detailed configuration example of the free viewpoint image generating device. [Diagram 3] 11 is a flowchart illustrating a free viewpoint image generating process executed by the free viewpoint image generating device. [Figure 4] 1 is a block diagram showing an example of the configuration of an embodiment of a computer to which the present technology is applied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, a description will be given of an embodiment of the present technology with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted. The description will be given in the following order. 1. Example of satellite image processing system configuration 2. Volumetric Capture Technology 3. Example of the configuration of a free viewpoint image generation device 4. Free viewpoint image generation processing 5. Application examples 6. Computer configuration example

[0015] <1. Example of satellite image processing system configuration> FIG. 1 is a block diagram showing an example of the configuration of a satellite image processing system according to an embodiment to which the present technology is applied.

[0016] The satellite image processing system 1 in Fig. 1 is a system that can observe any point on the ground at any time from any viewpoint in the sky using images (hereinafter referred to as satellite images) captured by multiple artificial satellites (hereinafter simply referred to as satellites). In this embodiment, the satellite is equipped with an imaging device and has at least the function of capturing images of the ground.

[0017] The satellite operating company has a satellite management device 11 that manages multiple satellites 21, and multiple communication devices 13 that communicate with the satellites 21. Note that the satellite management device 11 and some of the multiple communication devices 13 may be devices owned by parties other than the satellite operating company. The satellite management device 11 and the multiple communication devices 13 are connected via a predetermined network 12. The communication devices 13 are disposed in a ground station (terrestrial base station) 15. Note that, although an example is shown in FIG. 1 in which the number of communication devices 13 is three, communication devices 13A to 13C, the number of communication devices 13 is arbitrary.

[0018] The satellite management device 11 manages a plurality of satellites 21 owned by a satellite operating company. Specifically, the satellite management device 11 acquires related information from one or more information providing servers 41 of external organizations as necessary, and determines an operation plan for the plurality of satellites 21 owned by the satellite management device 11. Then, the satellite management device 11 instructs a specific satellite 21 to capture an image via the communication device 13 in response to a customer's request, thereby causing the specific satellite 21 to capture an image. The satellite management device 11 also acquires and stores satellite images transmitted from the satellite 21 via the communication device 13. The acquired satellite images are subjected to predetermined image processing as necessary, and are provided (transmitted) to the customer. The acquired satellite images are also provided (transmitted) to a free viewpoint image generating device 51 of the image providing company.

[0019] An information providing server 41 installed at an external organization supplies predetermined related information to the satellite management device 11 via a predetermined network in response to a request from the satellite management device 11 or periodically. Examples of related information provided by the information providing server 41 include the following. For example, satellite orbit information described in TLE (Two Line Elements) format can be obtained as related information from NORAD (North American Aerospace Defense Command) as an external organization. Also, for example, meteorological information such as the weather and cloud cover at a predetermined point on the ground can be obtained from a weather information provider as an external organization.

[0020] The free viewpoint image generating device 51 executes image processing to generate a free viewpoint image, which is a satellite image observed from a free viewpoint in the sky at any time for any point on the ground, using a satellite image captured by a satellite 21 and supplied from the satellite management device 11 via a predetermined network. The free viewpoint image generating device 51 transmits a satellite image capturing request to the satellite management device 11 as necessary in response to a generation instruction from a user 91 (FIG. 2). The generation processing of the free viewpoint image may be performed by a satellite operating company, in which case the satellite operating company and the image providing company are the same. The satellite management device 11 and the free viewpoint image generating device 51 may be realized by a single device.

[0021] The communication device 13 communicates with a specific satellite 21 designated by the satellite management device 11 via the antenna 14 under the control of the satellite management device 11. For example, the communication device 13 transmits an imaging instruction to the specific satellite 21 to image a specific area on the ground at a specific time and position. The communication device 13 also receives satellite images transmitted from the satellite 21 and supplies them to the satellite management device 11 via the network 12. Transmission from the communication device 13 of the ground station 15 to the satellite 21 is also called an uplink, and transmission from the satellite 21 to the communication device 13 is also called a downlink. The communication device 13 can communicate directly with the satellite 21, and can also communicate via a relay satellite 22. For example, a geostationary satellite is used as the relay satellite 22.

[0022] The network 12 and the network between the information providing server 41 or the free viewpoint image generating device 51 and the satellite management device 11 are any communication network, and may be a wired communication network, a wireless communication network, or both. In addition, the network 12 and the network between the information providing server 41 or the free viewpoint image generating device 51 and the satellite management device 11 may be configured as one communication network or multiple communication networks. These networks can be communication networks or communication paths of any communication standard, such as the Internet, a public telephone line network, a wide area communication network for wireless mobile devices such as a so-called 4G line or 5G line, a WAN (Wide Area Network), a LAN (Local Area Network), a wireless communication network that communicates in accordance with the Bluetooth (registered trademark) standard, a communication path for short-distance wireless communication such as NFC (Near Field Communication), a communication path for infrared communication, a communication network for wired communication in accordance with a standard such as HDMI (registered trademark) (High-Definition Multimedia Interface) or USB (Universal Serial Bus), etc.

[0023] The multiple satellites 21 may be operated individually (single satellite), or may be configured as a single satellite group with multiple satellites operated on a satellite group basis. In the example of Fig. 1, the illustration of the single satellite 21 is omitted, and satellites 21A and 21B configure a first satellite group 31A, and satellites 21C and 21D configure a second satellite group 31B. Note that, for simplicity, the example of Fig. 1 shows an example in which one satellite group 31 is configured by two satellites 21, but the number of satellites 21 that configure one satellite group 31 is not limited to two.

[0024] Systems that operate multiple satellites 21 as one unit (satellite group) include constellations and formation flights. A constellation is a system that deploys services uniformly mainly around the globe by launching many satellites into a single or multiple orbital planes. Each individual satellite that constitutes a constellation has a specific function, and multiple satellites are operated for the purpose of improving observation frequency. On the other hand, a formation flight is a system in which multiple satellites are deployed while maintaining their relative positional relationship in a small area of ​​about several kilometers. A formation flight can provide services that cannot be realized by a single satellite, such as high-precision three-dimensional measurement and speed detection of moving objects. In this embodiment, it does not matter whether the operation of the satellite group is a constellation or a formation flight.

[0025] When the communication device 13 communicates with each satellite 21 constituting the satellite group 31, there are two methods: one is to communicate with each satellite 21 individually as in the first satellite group 31A in FIG. 1; and the other is to communicate with the communication device 13 indirectly through inter-satellite communication with the representative satellite 21C as in the second satellite group 31B. The method of communication with the earth station 15 (the communication device 13) may be determined in advance by the satellite group 31, or may be appropriately selected according to the content of the communication. The satellite 21 operated alone may communicate with the communication device 13 of the earth station 15 in some cases, or may communicate with the communication device 13 of the earth station 15 via a relay satellite 22 in other cases. When the satellite 21 moves within the communication range of the communication device 13 installed in the earth station 15, the satellite 21 can downlink with the communication device 13 at a predetermined frequency. When the satellite 21 is moving outside the communication range of the communication device 13 installed in the ground station 15, the satellite 21 can transmit the image data to another satellite 21 located within the communication range of the communication device 13 installed in the ground station 15 by inter-satellite communication, and downlink the image data to the ground station 15 via the other satellite 21. This ensures that the satellite images are available in real time.

[0026] The satellite image processing system 1 is configured as described above.

[0027] The satellite management device 11 or the free viewpoint image generating device 51 can perform the following image processing on the satellite images captured by the individual satellites 21.

[0028] (1) Metadata generation Metadata can be generated based on information transmitted from the satellite 21 and information about the satellite 21 that has taken the image. For example, information on the latitude and longitude of the position of the image subject, and information on the attitude control and acceleration when the satellite 21 took the image, can be generated as metadata. Note that the satellite 21 may generate metadata itself based on the conditions at the time of image capture, in which case metadata that has been added in advance to the satellite image taken by the satellite 21 may be used. (2) Satellite image correction processing Correction processes that can be performed include radiometric correction related to sensitivity characteristics, geometric correction of the orbital position and attitude error of the satellite 21, ortho-correction to correct geometric distortion caused by differences in elevation of the terrain, and map projection to project an image onto a map projection surface. (3) Color composition processing It is possible to perform color synthesis processes such as pan-sharpening, true color synthesis, false color synthesis, natural color synthesis, SAR image synthesis, and processing to add color to satellite images for each band. (4) Other image synthesis It is also possible to combine satellite images captured by the satellite 21 in the past, satellite images captured by other satellites 21, or other images, combine satellite images captured in different bands, and combine them with map information. (5) Information Extraction Using different bands such as R (Red) and IR (Infrared), it is possible to calculate vegetation detection information such as NDVI (Normalized Difference Vegetation Index) and water detection information such as NDWI (Normalized Difference Water Index). It is possible to perform highlight processing of specific subjects such as vehicles, moving objects, and schools of fish, and to extract information on specific bands and changes from the previous image capture.

[0029] In particular, when using multiple satellite images captured by multiple satellites 21 performing a constellation or formation flight, the satellite management device 11 or the free viewpoint image generation device 51 can more effectively perform image processing such as the following:

[0030] (1) High-resolution or high-quality processing By overlaying multiple satellite images, it is possible to generate satellite images with improved resolution. In addition, it is possible to generate pan-sharpened images by combining monochrome and color images, or high-resolution satellite images by synthesizing satellite images taken under different conditions, such as different dynamic ranges, shutter speeds, different bands (wavelength bands), and different resolutions. (2) Division of Functions Different bands such as R (Red) and IR (Infrared) can be used to calculate indices such as NDVI (Normalized Difference Vegetation Index). (3) 3D measurement Parallax images can provide three-dimensional information. In addition, three-dimensional information can improve the accuracy of object recognition on the ground. For example, it can be used to determine whether an object is a vehicle or not (even if the resolution of the image does not immediately indicate that it is a vehicle, if it is determined that the object on the road is a three-dimensional object rather than a pattern, it can be assumed that it is a vehicle). (4) Differential measurement Using multiple satellite images taken at different times from the same position, it is possible to extract the change between the first time and the second time. It is also possible to perform imaging by extracting and coloring only the changed objects. For example, using multiple satellite images, it is possible to calculate the moving speed of ships and vehicles, and to calculate wind speed from the movement of clouds, etc. (5) Other image synthesis It is also possible to combine past satellite images, satellite images captured by other satellites 21, satellite images captured in different bands, and images with map information.

[0031] Each satellite 21 may transmit satellite images of the ground to the communication device 13 as RAW data, or may transmit the data after undergoing the above-mentioned image processing.

[0032] (Image format) The processed images after image processing and the satellite images are stored in the storage unit of each device using, for example, the following image format, and are transmitted to other devices. (1) CEOS CEOS is a format standardized by the Committee on Earth Observation Satellites. There are two types of CEOS: "CEOS-BSQ", in which the files are divided by band, and "CEOS-BIL", in which multiple bands are multiplexed. (2) HDF This format was developed by the National Center for Supercomputing Applications (NCSA) at the University of Illinois. Multiple bands are packed into one file to facilitate data interchange among various computer environments. (3) Geo TIFF This is a format that adds remote sensing information to TIFF (Tagged Image File Format). Since it is in TIFF format, it can be opened with a general image viewer. (4)JPEG2000 An image format standardized by the Joint Photographic Experts Group. JPEG 2000 does not simply increase the compression rate, but also incorporates techniques for improving images in the area of ​​interest and copyright protection techniques such as digital watermarking.

[0033] <2. Volumetric Capture Technology> There is a known technology that generates a 3D model of a subject from images (including video) captured from multiple viewpoints, and generates a virtual viewpoint image of the 3D model according to an arbitrary viewing position, thereby providing an image from any viewpoint. This technology is also called volumetric capture technology.

[0034] The free viewpoint image generating device 51 applies volumetric capture technology to satellite images, and generates a free viewpoint image, which is a satellite image of an arbitrary point on the ground observed at an arbitrary time from a free viewpoint in the sky.

[0035] First, a brief description will be given of the generation of a 3D model of a subject when the subject is a person or the like, and the display of a free viewpoint image based on the generated 3D model.

[0036] In generating a 3D model, a plurality of captured images are obtained by capturing images of a predetermined shooting space in which a subject is placed from the periphery of the space using a plurality of image capturing devices. The captured images are, for example, composed of a moving image.

[0037] Using images captured by multiple imaging devices in different directions, a foreground subject to be displayed in the captured space is extracted, and a 3D object, which is a 3D model of the subject, is generated (3D modeling). For example, a 3D object is generated by projecting the silhouette of the subject from each viewpoint into a 3D space, using Visual Hull, which creates a 3D shape by projecting the silhouette intersection area, or by Multi-view stereo, which utilizes the consistency of texture information between viewpoints.

[0038] Then, data of one or more 3D objects (hereinafter also referred to as 3D model data) among one or more 3D objects existing in the shooting space is transmitted to a playback device and played back. That is, the playback device renders the 3D objects based on the acquired 3D object data, and displays the 3D shape image on the viewer's viewing device. The viewing device is, for example, a liquid crystal display, a head-mounted display, or the like.

[0039] The 3D model data may have a variety of data formats.

[0040] One of the data formats represents the geometric information (shape information) of an object by expressing the object's three-dimensional position as a set of points (point cloud), and holds the color information of the object corresponding to each point. In this format, one piece of geometric information and color information is held for one object. This format is described as the point cloud format.

[0041] Another data format is a format in which the geometry information of an object is represented as a set of points (point cloud) similar to the point cloud format described above, or as vertices (vertices) and connections between the vertices, called a polygon mesh, and the color information of the object is held in the captured images (two-dimensional texture images) captured by each imaging device. In this format, one piece of geometry information and color information consisting of the same number of captured images (two-dimensional texture images) as the number of imaging devices are held for one object. This format is described as a multi-texture geometry format.

[0042] Yet another data format is a format in which the geometry information of an object is represented by a polygon mesh, and color information of the object is held corresponding to each polygon mesh. A two-dimensional texture image as color information to be applied to each polygon mesh is expressed in a UV coordinate system. In this format, one piece of geometry information and color information consisting of one two-dimensional texture image are held for one object. In this embodiment, this format is referred to as a UV texture geometry format. The UV texture geometry format is a format standardized by MPEG-4 AFX (Animation Framework eXtension).

[0043] Yet another data format is a format in which the geometry information of an object is represented by distance information corresponding to the photographed image captured by each imaging device, and the color information of the object is held in the photographed image (two-dimensional texture image) captured by each imaging device. For the distance information corresponding to the photographed image captured by each imaging device, a depth image is adopted in which the distance in the depth direction to the subject is stored as a depth value in correspondence with each pixel of the photographed image. In this format, for one object, geometry information consisting of depth images in the same number as the number of imaging devices and color information consisting of photographed images (two-dimensional texture images) in the same number as the number of imaging devices are held. This format is described as a multi-texture depth format. The advantage of the multi-texture depth format is that it can directly use the AVC (Advanced Video Coding) format, the HEVC (High Efficiency Video Coding) format, etc. as an encoding format when transmitting 3D model data, and can compress with high efficiency.

[0044] Of the various data formats for 3D model data as described above, any data format can be adopted. The playback side may specify the data format, or the distribution side may determine the data format. Also, a data format may be determined in advance for each application.

[0045] The playback side can request only the 3D object to be viewed among one or more 3D objects existing in the shooting space, and display it on the viewing device. For example, the playback side assumes a virtual camera whose shooting range is the viewing range of the viewer, and requests only the 3D object captured by the virtual camera among many 3D objects existing in the shooting space, and displays it on the viewing device. The viewpoint (virtual viewpoint) of the virtual camera can be set to any position so that the viewer can view the subject from any viewpoint in the real world. A background image representing a predetermined space is appropriately synthesized with the 3D object. The background image may be a still image with a fixed virtual viewpoint, or may be an image that changes according to the virtual viewpoint, like the subject as the foreground image.

[0046] <3. Example of the configuration of a free viewpoint image generation device> FIG. 2 is a block diagram showing an example of a detailed configuration of the free viewpoint image generating device 51. As shown in FIG.

[0047] The free viewpoint image generating device 51 includes a free viewpoint image generating unit 61, a free viewpoint image storage unit 62, an encoding unit 63, a communication unit 64, and a user IF (Interface) unit 65. The user IF unit 65 includes a display unit 81 and an operation unit .

[0048] The free viewpoint image generating device 51 can receive an operation of the user 91 via the user IF unit 65 and execute processing according to the instruction of the user 91, and can also execute processing according to the instruction when the user 91 instructs the free viewpoint image generating device 51 to execute a specific process via the terminal device 92. The user 91 instructs the generation of a free viewpoint image, which is a satellite image of a specific point on the ground (specified point) viewed at a specific time (specified time) from a specific virtual viewpoint in the sky, by directly inputting to the operation unit 82 or via the terminal device 92.

[0049] In response to an instruction from a user to generate a free viewpoint image, the free viewpoint image generating device 51 generates a satellite image of a specified point on the ground viewed from a specified virtual viewpoint at a specified time by combining a base image viewed from a specified virtual viewpoint and a real-time image.

[0050] The base image seen from a specific virtual viewpoint is a satellite image equivalent to the background image in volumetric capture technology, and is a satellite image of a stationary subject that shows no change over a certain time span (changes over a certain time span are not an issue). The base image is an image that shows relatively little change compared to the real-time image.

[0051] In contrast, a real-time image seen from a predetermined virtual viewpoint is a satellite image that corresponds to the foreground image (subject) in volumetric capture technology, and is a satellite image of a dynamic subject (real-time subject) that changes in real time.

[0052] A dynamic subject that changes in real time (real-time subject) is a subject that changes over a certain period of time other than a still subject included in the base image, and includes, for example, subjects that change in an instant as well as subjects that change over the course of a few hours or a day.

[0053] Dynamic subjects include, for example, moving objects such as airplanes, ships, vehicles, and people; meteorological phenomena such as clouds, auroras, and volcanic eruptions; the reflection of the sun on the ocean, lakes, rivers, and earth's surface; the color of the sky at sunrise or sunset, and light information including shadows.

[0054] Subjects that change over a period of time longer than a day, other than dynamic subjects, are considered to be still subjects. For example, the changing colors of mountains, such as autumn leaves, the state of rice paddies immediately after planting, and the state of rice ears at harvest time are seen as seasonal changes, so mountain forests and rice fields are considered to be still subjects.

[0055] The free viewpoint image generating unit 61 has a base image generating unit 71 , a real-time image generating unit 72 , an external information acquiring unit 73 , and an image synthesizing unit 74 .

[0056] The free viewpoint image generating unit 61 of the free viewpoint image generating device 51 is connected to the satellite image accumulation server 101 and the external information providing server 102 via a predetermined network. The network between the free viewpoint image generating device 51 and the satellite image accumulation server 101 or the external information providing server 102 can be any communication network similar to the network 12 described above. The connection to the predetermined network is made via a communication unit 64.

[0057] The free viewpoint image generating unit 61 generates a free viewpoint image of a predetermined point on the ground viewed from a predetermined virtual viewpoint in the sky based on a generation instruction from the control unit 66 , and supplies the generated image to the free viewpoint image storage unit 62 .

[0058] The base image generating unit 71 acquires a plurality of satellite images captured from different viewpoints that are accumulated (stored) in the satellite image accumulation server 101, and generates a 3D model of a still subject on the ground using the acquired plurality of satellite images. Ta A first virtual viewpoint image when a 3D model of a stationary subject on the ground is viewed from a predetermined virtual viewpoint is generated, and is supplied to an image synthesis unit 74 as a base image.

[0059] The base image generating unit 71 can use the satellite images acquired from the satellite image storage server 101 to generate a 3D model of a still subject on the ground after removing dynamic subjects included in the satellite images acquired from the satellite image storage server 101 in advance. In other words, the base image generating unit 71 generates a satellite image from the satellite images acquired from the satellite image storage server 101, from which subjects to be included in the real-time image generated by the real-time image generating unit 72 have been removed, and the base image generating unit 71 can use the satellite images to generate a 3D model of a still subject on the ground. Removal of dynamic subjects included in satellite images can be achieved, for example, by comparing satellite images captured at the same point, detecting subjects that do not match as dynamic subjects, and removing them. Also, for example, minute changes can be detected as dynamic subjects using satellite images captured by multiple artificial satellites such as a formation flight, and removing them.

[0060] The satellite image storage server 101 stores a plurality of satellite images captured from the sky by an artificial satellite equipped with an imaging device. The artificial satellite may be a satellite 21 of the satellite operating company that owns the satellite management device 11, or may be an artificial satellite of another company. The satellite image storage server 101 may be a server operated by the satellite operating company that owns the satellite management device 11, may be a server operated by the image providing company that owns the free viewpoint image generation device 51, or may be a server operated by another company. The satellite image storage server 101 may be included as a part of the free viewpoint image generation device 51 as a satellite image storage unit, and may store satellite images provided by the satellite management device 11.

[0061] The satellite images stored in the satellite image storage server 101 are images obtained by capturing images of the ground from multiple viewpoints in the sky using one or multiple artificial satellites. When capturing images using one artificial satellite, multiple satellite images corresponding to the multiple viewpoints are generated by capturing images of the ground in a time-division manner. When multiple artificial satellites are used, multiple satellite images corresponding to multiple viewpoints can be generated by each capturing images of the ground in a time-division manner.

[0062] The satellite images stored in the satellite image storage server 101 may be two satellite images captured by an artificial satellite combined into one. For example, two satellite images with partially overlapping imaging areas can be stitched together to create one satellite image. The two satellite images to be stitched may have different resolutions.

[0063] Instead of the satellite images stored in the satellite image storage server 101, aerial images taken by an aircraft may be used.

[0064] It is desirable that the resolution of the satellite images stored in the satellite image storage server 101 is high because the accuracy of subsequent processing changes. For example, it is desirable that the ground resolution is 1 m or less, and if it is desired to identify ground structures such as vehicles, it is preferable that the resolution is 50 cm or less.

[0065] The satellite image storage server 101 may store depth images, which are depth information to still subjects, in addition to satellite images as texture information. Depth images can be generated, for example, from parallax information based on multiple satellite images taken of the same point from different viewpoints by a single or multiple satellites. Imaging by multiple satellites includes imaging by formation flight. Depth images may also be generated from altitude measurement results by synthetic aperture radar satellites (SAR satellites). Alternatively, the depth images may be generated by estimation from two-dimensional satellite images, such as estimating height information based on the size of a shadow reflected in a satellite image.

[0066] The satellite images stored in the satellite image storage server 101 can be stored for different times such as morning, noon, afternoon, evening, and night, different weather conditions such as clear, sunny, cloudy, and rainy, and different conditions for each season such as spring, summer, autumn, and winter. In this case, the base image generating unit 71 can generate a 3D model of a still subject on the ground for each stored condition. If a satellite image of a predetermined condition is not stored in the satellite image storage server 101, the base image generating unit 71 can estimate the satellite image based on a satellite image of the same point captured under other conditions, generate a satellite image that is not stored, and use it in the generation process of the 3D model of the still subject. For example, if an autumn satellite image is not stored, an autumn satellite image may be estimated and generated from a satellite image of the same point in summer or winter.

[0067] The satellite image storage server 101 may store 3D model data of a ground stationary subject generated by 3D modeling in another device, instead of satellite images captured from multiple viewpoints in the sky. In this case, the base image generating unit 71 acquires 3D model data of a ground stationary subject stored in the satellite image storage server 101, and generates a first virtual viewpoint image based on the acquired 3D model data of the ground stationary subject. In other words, when 3D model data of a ground stationary subject is stored in the satellite image storage server 101, the 3D modeling process is omitted.

[0068] The data format of the 3D model of the ground stationary subject stored in the satellite image storage server 101 may be any of the data formats of the 3D model data described above. The data format when the base image generating unit 71 generates the 3D model of the ground stationary subject may also be any of the formats.

[0069] Since the base image is a satellite image of a still subject that is not affected by the designated time of the free viewpoint image designated by the user 91, the satellite image stored in the satellite image storage server 101 can be a satellite image captured about one week or one month before the designated time of the free viewpoint image designated by the user 91. Alternatively, the satellite image stored in the satellite image storage server 101 may be a satellite image captured after (in the future) the designated time of the free viewpoint image designated by the user 91.

[0070] There may be cases where a satellite image captured at a newer time or with a higher resolution than a satellite image acquired from the satellite image storage server 101 is supplied from the real-time image generating unit 72. When a more recent or higher resolution satellite image is supplied from the real-time image generating unit 72, the base image generating unit 71 can use the satellite image to update the 3D model of the still subject on the ground and generate a base image.

[0071] The real-time image generating unit 72 acquires from the satellite management device 11 a plurality of satellite images captured from different viewpoints, which correspond to a specified point and a specified time specified by the user 91 as the point and time for generating a free viewpoint image. The plurality of satellite images from different viewpoints corresponding to the specified time are called real-time satellite images, to distinguish them from the satellite images used for generating a base image.

[0072] The real-time image generating unit 72 supplies the satellite management device 11 with the designated point and the designated time of the free viewpoint image, and acquires a plurality of satellite images (real-time satellite images) captured at the designated time from different viewpoints. The satellite management device 11 transmits an image capturing instruction from the communication device 13 to the satellite 21 passing the designated point in the vicinity of the designated time supplied from the real-time image generating unit 72, and causes the satellite 21 to capture the image. The vicinity of the designated time may be a time including an error of several minutes to several tens of minutes. The satellite management device 11 or the real-time image generating unit 72 may change the captured satellite image to an image that estimates a change according to the time error. For example, when an airplane flying at a specified point at a specified time is captured as a dynamic subject, if the time has an error with respect to the specified time, the image may be changed to a satellite image in which the position of the airplane has been moved by the time of the error.

[0073] A plurality of satellite images from different viewpoints may be captured by one satellite 21 in a time-division manner, or may be captured by a plurality of satellites 21 having different orbits from their respective viewpoints. Alternatively, a plurality of satellites 21 having the same orbit, or a plurality of satellites 21 operating in a formation flight, may capture images with a time difference of about several minutes to several tens of minutes. For satellite images with a time difference of about several minutes to several tens of minutes, the satellite image at the time closest to the specified time may be used as a reference, and the other satellite images may be changed to images that estimate changes according to time errors.

[0074] The real-time image generating unit 72 extracts dynamic subjects on the ground from each of the multiple acquired real-time satellite images. The dynamic subjects on the ground can be extracted, for example, by subtracting the images from a base image of the same viewpoint. Clouds, airplanes, ships, vehicles, and the like have characteristic shapes and colors, so dynamic subjects may be extracted by image recognition processing based on these characteristics. Alternatively, dynamic subjects may be extracted using the same processing as that used by the base image generating unit 71 to remove dynamic subjects.

[0075] The real-time image generating unit 72 generates a 3D model of the dynamic subject from each of the real-time satellite images including only the extracted dynamic subject on the ground. The real-time image generating unit 72 then generates a second virtual viewpoint image as seen when the 3D model of the dynamic subject is viewed from a predetermined virtual viewpoint designated by the user 91 as a real-time image of the dynamic subject, and supplies the second virtual viewpoint image to the image synthesis unit 74.

[0076] Instead of generating a 3D model of a dynamic subject, the real-time image generating unit 72 may generate three-dimensional information based on a parallax image calculated from a plurality of satellite images capturing a specified point from different viewpoints. Alternatively, instead of generating a 3D model of a dynamic subject, the real-time image generating unit 72 may generate three-dimensional information by estimation using only two-dimensional satellite images without calculating depth information. For example, an airplane may be extracted as a dynamic subject from a two-dimensional satellite image, and three-dimensional information of the dynamic subject may be generated from the flight altitude of the airplane as known information.

[0077] The data format in which the real-time image generating unit 72 generates a 3D model of a dynamic subject may be any of the above-mentioned data formats of the 3D model data. If a 3D model is not formed, only a two-dimensional satellite image is stored in the internal memory.

[0078] When the real-time image generation unit 72 acquires a real-time satellite image from the satellite management device 11, it can supply the acquired real-time satellite image to the base image generation unit 71 as an update satellite image for updating the satellite image of the same location.

[0079] If there is a satellite 21 passing the specified point near the specified time supplied from the real-time image generating unit 72, then the satellite 21 should be made to take an image, but in reality, there may be cases where there is no satellite 21 passing the specified point near the specified time. In such a case, the real-time image generating unit 72 generates a real-time image of the dynamic subject using external information acquired by the external information acquiring unit 73 from the external information providing server 102, and supplies the image to the image synthesizing unit 74.

[0080] When the dynamic subject is a moving object such as an airplane, a ship, a vehicle, or a person, the position information of the moving object at the specified time is acquired from the external information providing server 102 based on the information of the specified point and the specified time of the free viewpoint image. For example, for a ship, an airplane, etc., the position information of the moving object at the specified time can be acquired by acquiring AIS (Automatic Identification System) information from the external information providing server 102. For a vehicle, a person, etc., the position information of the moving object at the specified time can be acquired by acquiring the position information of a device attached to the vehicle or person or the position information detected by a surveillance camera installed on the ground from the external information providing server 102. The position information of various moving objects may be acquired from the external information providing server 102 operated by an operating company that provides the position information of the moving object.

[0081] The real-time image generation unit 72 acquires known 3D models or texture information of moving objects from an external company or generates it internally and stores it in advance. Based on the position information of the moving object acquired via the external information acquisition unit 73, a two-dimensional image of the moving object is placed at a predetermined position, thereby generating a real-time image of the dynamic subject and supplying it to the image synthesis unit 74.

[0082] When the dynamic subject is a meteorological phenomenon such as cloud distribution, a real-time image of cloud distribution can be generated by acquiring information indicating cloud distribution and estimated altitude information from an external information providing server 102 operated by a weather service company that provides weather information, and arranging a two-dimensional image of the cloud at the estimated altitude. Also, for example, when the dynamic subject is a solar reflection phenomenon, a real-time image of light ray information such as a shadow or solar reflection can be generated by acquiring information such as the sun position from an external information providing server 102 operated by a weather service company, estimating light ray information, and arranging it at a predetermined position.

[0083] For ease of explanation, only one external information providing server 102 is shown in Figure 2, but the external information acquisition unit 73 can acquire the desired external information by accessing external information providing servers 102 from different external information providers or in different locations depending on the type of external information to be acquired.

[0084] In the above description, the real-time image generating unit 72 generates a real-time image of a dynamic subject for each of the cases where it is possible and impossible to capture a satellite image corresponding to the designated point and designated time designated by the user 91, but in reality, there may be intermediate cases between these. That is, while it is not possible to capture a satellite image of a designated point within an error range of about several tens of minutes from the designated time designated by the user 91, there are cases where it is possible to obtain a satellite image of a designated point within an error range of one hour, three hours, six hours, or several hours from the designated time.

[0085] In this case, the real-time image generating unit 72 acquires from the satellite management device 11 a plurality of satellite images captured from different viewpoints at times within a predetermined time error range from the designated time specified by the user 91. The plurality of satellite images from different viewpoints corresponding to times within a predetermined time error range from the designated time are called near-real-time satellite images to distinguish them from the satellite images used to generate the base image and the real-time satellite images. An error range of several hours from the designated time is the error range assumed in the operation of a constellation, but the error range of the near-real-time satellite images from the designated time is a maximum of one day assuming an observation satellite with a recurrence period of one day.

[0086] The real-time image generating unit 72 extracts a dynamic subject on the ground from each of the acquired multiple near real-time satellite images and generates a 3D model of the dynamic subject. The extraction of the dynamic subject and the generation of the 3D model of the dynamic subject are similar to the case of the real-time satellite image described above.

[0087] Next, the real-time image generating unit 72 estimates a 3D model of the dynamic subject at a specified time from the generated 3D model of the dynamic subject using external information related to the dynamic subject acquired from the external information providing server 102 via the external information acquiring unit 73. For example, the real-time image generating unit 72 acquires position information of an airplane or a ship as external information, and moves the 3D model of the moving object to a position based on the external information. Also, for example, the real-time image generating unit 72 acquires cloud distribution information as external information, and moves the 3D model of the cloud to a position based on the external information. Then, the real-time image generating unit 72 generates a second virtual viewpoint image when the 3D model of the dynamic subject at the specified time estimated based on the external information is viewed from a predetermined virtual viewpoint designated by the user 91 as a real-time image of the dynamic subject, and supplies the second virtual viewpoint image to the image synthesis unit 74.

[0088] Alternatively, the real-time image generating unit 72 generates a quasi-real-time image of the dynamic subject when the 3D model of the dynamic subject generated from the multiple acquired quasi-real-time satellite images is viewed from a predetermined virtual viewpoint designated by the user 91. The real-time image generating unit 72 estimates and generates a real-time image of the dynamic subject using the generated quasi-real-time image of the dynamic subject and external information related to the dynamic subject acquired from the external information providing server 102 via the external information acquiring unit 73. For example, a quasi-real-time image that is a texture image of an airplane or ship from two hours ago is changed to a position at a specified time based on the external information, and becomes a real-time image of the moving object at the specified time. A quasi-real-time image that is a texture image of clouds from two hours ago is changed based on information on clouds at a specified time based on the external information, and a real-time image of clouds at a specified time is generated.

[0089] In other words, when near real-time satellite images are acquired, a real-time image of a dynamic subject is generated by linking (mutually complementing) both a method of acquiring real-time satellite images to generate a real-time image of a dynamic subject, and a method of generating a real-time image of a dynamic subject using only external information.

[0090] Even if a real-time satellite image corresponding to the designated time designated by the user 91 can be acquired, a quasi-real-time satellite image may also be used. By using a real-time satellite image and a quasi-real-time satellite image in combination, the following effects can be obtained. As one effect, an image with a small amount of information of a real-time satellite image captured at a designated time can be tolerated. For example, by using both a real-time satellite image and a quasi-real-time satellite image, the resolution of the real-time satellite image captured at a designated time can be low, and the imaging range can be expanded by the amount of the low resolution. Different types of satellites 21 can be used for the satellites 21 for capturing real-time satellite images and the satellites 21 for capturing quasi-real-time satellite images, and there is room to prepare multiple types of satellites 21. Based on the difference information between the real-time satellite image and the quasi-real-time satellite image, information during that time, such as changes in clouds during that time period, can also be supplemented.

[0091] In addition, the real-time image generating unit 72 may estimate light ray information as a dynamic subject using a base image that can be generated by the base image generating unit 71, and generate a real-time satellite image of the light ray information. For example, the base image generating unit 71 generates a 3D model of a still subject under each condition, such as different times of day, such as morning, noon, and night, clear and sunny weather, and seasons, such as spring, summer, autumn, and winter, and can generate a base image for each condition. The real-time image generating unit 72 detects changes in light ray information in the base image for each condition generated from the 3D model of the still subject for each condition, and can generate a real-time satellite image based on the time, season, and weather specified by the user 91 by estimation. The real-time satellite image may be estimated using a quasi-real-time satellite image or an incomplete real-time satellite image. The incomplete real-time satellite image is, for example, a panchromatic image (monochrome image) or an image with low resolution. Texture information representing the shadow and color of a moving object can also be estimated using the base image, similar to the light ray information.

[0092] The image synthesis unit 74 synthesizes the base image supplied from the base image generation unit 71 and the real-time satellite image supplied from the real-time image generation unit 72 to generate a free viewpoint image of a specified point on the ground at a specified time specified by the user 91, and supplies the generated image to the free viewpoint image storage unit 62. The image synthesis unit 74 superimposes the real-time image corresponding to the foreground image (subject) of the volumetric capture technology on the base image corresponding to the background image of the volumetric capture technology.

[0093] The free viewpoint image storage unit 62 stores the free viewpoint images supplied from the image synthesis unit 74. The free viewpoint image storage unit 62 stores free viewpoint images of various virtual viewpoints, designated times, and designated points generated by the free viewpoint image generation unit 61, and can select a designated free viewpoint image in response to an instruction from the control unit 66 and supply it to the encoding unit 63 or the display unit 81.

[0094] The encoding unit 63 encodes the free viewpoint image supplied from the free viewpoint image storage unit 62 using a predetermined encoding method such as the Advanced Video Coding (AVC) method or the High Efficiency Video Coding (HEVC) method. The encoded free viewpoint image encoded data is supplied to the communication unit 64.

[0095] The communication unit 64 communicates with the terminal device 29 via a predetermined network. When a generation instruction for generating a free viewpoint image of a predetermined virtual viewpoint, a specified time, and a specified point is supplied from the terminal device 92, the communication unit 64 supplies the generation instruction to the control unit 66. In response to the generation instruction, the communication unit 64 transmits free viewpoint image encoded data supplied from the encoding unit 63 to the terminal device 92. In addition, the communication unit 64 transmits a satellite image capture request to the satellite management device 11 under the control of the control unit 66. The communication unit 64 also communicates with the satellite image storage server 101 and the external information providing server 102 under the control of the control unit 66.

[0096] The display unit 81 is configured with, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display. The display unit 81 displays the free viewpoint image supplied from the free viewpoint image accumulation unit 62.

[0097] The operation unit 82 is composed of, for example, a keyboard, a mouse, a touch panel, etc., and accepts operations from the user 91 and supplies a generation instruction to the control unit 66 to generate a free viewpoint image of the virtual viewpoint, specified time, and specified point specified by the user 91.

[0098] The control unit 66 controls the operation of the entire free viewpoint image generating device 51. For example, the control unit 66 supplies the free viewpoint image generating unit 61 with a generation instruction for a free viewpoint image of a predetermined point on the ground viewed at a predetermined time from a predetermined virtual viewpoint in the sky specified by the user 91 based on a generation instruction from the communication unit 64 or the user IF unit 65. In addition, when it is necessary to capture a satellite image required for generating the free viewpoint image, the control unit 66 causes the communication unit 64 to transmit a satellite image capture request to the satellite management device 11. In addition, the control unit 66 causes the free viewpoint image stored in the free viewpoint image storage unit 62 to be supplied to the encoding unit 63 or the display unit 81.

[0099] The terminal device 92 is composed of, for example, a smartphone, a tablet terminal, a mobile phone, a personal computer, or the like, and accepts an operation of the user 91. When the user 91 instructs the free viewpoint image generating device 51 to generate a free viewpoint image via the terminal device 92, the terminal device 92 transmits the generation instruction to the free viewpoint image generating device 51. In addition, the terminal device 92 receives free viewpoint image encoded data transmitted from the free viewpoint image generating device 51 in response to the generation instruction, performs a decoding process corresponding to the encoding method, and displays the free viewpoint image on a display device (not shown). The user can check the free viewpoint image from any location (remotely) using the terminal device 92 owned by the user.

[0100] <4. Free viewpoint image generation processing> Next, a free viewpoint image generation process executed by the free viewpoint image generation device 51 will be described with reference to the flowchart in Fig. 3. This process is started when, for example, the user 91 performs an operation on the terminal device 92 to instruct generation of a free viewpoint image in which a specified specified point on the ground is viewed at a specified specified time from a specified virtual viewpoint in the sky.

[0101] First, in step S1, the control unit 66 transmits an imaging request for a satellite image corresponding to a specific virtual viewpoint, a specified time, and a specified point to the satellite management device 11 via the communication unit 64 based on a generation instruction from the terminal device 92. The control unit 66 also supplies an instruction to the free viewpoint image generation unit 61 to generate a free viewpoint image of a specific point on the ground viewed at a specific time from a specific virtual viewpoint in the sky specified by the user 91.

[0102] In step S2, the base image generating unit 71 acquires a plurality of satellite images captured from different viewpoints and stored in the satellite image storage server 101, and removes moving subjects contained in each of the plurality of satellite images.

[0103] In step S3, the base image generating unit 71 generates a 3D model of a still subject on the ground using the multiple satellite images from which the dynamic subjects have been removed. Then, in step S4, the base image generating unit 71 generates a first virtual viewpoint image as a base image when the generated 3D model of the still subject is viewed from a predetermined virtual viewpoint designated by the user 91, and supplies the first virtual viewpoint image to the image synthesis unit 74.

[0104] In step S5, the real-time image generation unit 72 determines whether or not multiple real-time satellite images captured from different viewpoints corresponding to the specified location and specified time specified by the user 91 have been obtained from the satellite management device 11 in response to the imaging request in step S1.

[0105] If it is determined in step S5 that multiple real-time satellite images captured from different viewpoints have been acquired, the process proceeds to step S6, and the real-time image generation unit 72 extracts a dynamic subject from each of the multiple acquired real-time satellite images.

[0106] In step S7, the real-time image generating unit 72 generates a 3D model of the dynamic subject from each of the real-time satellite images including only the extracted dynamic subject on the ground. Then, in step S8, the real-time image generating unit 72 generates a second virtual viewpoint image when the 3D model of the dynamic subject is viewed from a predetermined virtual viewpoint designated by the user 91 as a real-time image of the dynamic subject, and supplies the second virtual viewpoint image to the image synthesis unit 74.

[0107] On the other hand, if it is determined in step S5 that multiple real-time satellite images taken from different viewpoints could not be obtained, the processing proceeds to step S9, and the real-time image generating unit 72 determines whether multiple quasi-real-time satellite images taken from different viewpoints, which are satellite images taken at a time within a specified time error range from the specified time specified by the user 91, could be obtained from the satellite management device 11.

[0108] If it is determined in step S9 that multiple quasi-real-time satellite images taken from different viewpoints have been acquired from the satellite management device 11, the processing proceeds to step S10, and the real-time image generation unit 72 extracts dynamic subjects from each of the multiple acquired quasi-real-time satellite images, and generates a 3D model of the dynamic subjects from each of the extracted quasi-real-time satellite images that include only the dynamic subjects on the ground.

[0109] In step S11 , the real-time image generating unit 72 acquires external information relating to the dynamic subject from the external information providing server 102 via the external information acquiring unit 73 .

[0110] In step S12, the real-time image generating unit 72 estimates a 3D model of the dynamic subject at the specified time from the generated 3D model of the dynamic subject, using the acquired external information related to the dynamic subject.

[0111] In step S13, the real-time image generation unit 72 generates a second virtual viewpoint image when the estimated 3D model of the dynamic subject at the specified time is viewed from a specific virtual viewpoint specified by the user 91 as a real-time image of the dynamic subject, and supplies it to the image synthesis unit 74.

[0112] On the other hand, if it is determined in step S9 that multiple quasi-real-time satellite images taken from different viewpoints cannot be obtained from the satellite management device 11, the processing proceeds to step S14, and the real-time image generation unit 72 obtains external information regarding the dynamic subject from the external information providing server 102 via the external information acquisition unit 73.

[0113] In step S15 , the real-time image generating unit 72 generates a real-time image of the dynamic subject using the acquired external information, and supplies the image to the image synthesizing unit 74 .

[0114] In step S16, the image synthesis unit 74 synthesizes the base image supplied from the base image generation unit 71 and the real-time satellite image supplied from the real-time image generation unit 72 to generate a free viewpoint image of a specified point on the ground at a specified time specified by the user 91, and supplies the free viewpoint image to the free viewpoint image storage unit 62.

[0115] In step S17, the free viewpoint image storage unit 62 stores the free viewpoint image supplied from the image synthesis unit 74, and supplies the free viewpoint image supplied from the image synthesis unit 74 to the encoding unit 63 under the control of the control unit 66.

[0116] In step S18, the encoding unit 63 encodes the free viewpoint image supplied from the free viewpoint image storage unit 62 using a predetermined encoding method, and the communication unit 64 transmits the encoded free viewpoint image encoded data to the terminal device 92. The free viewpoint image based on the free viewpoint image encoded data is displayed on the terminal device 92.

[0117] In addition, when a generation instruction is supplied from the operation unit 82, the processing of steps S17 and S18 is changed to processing in which the free viewpoint image supplied from the image synthesis unit 74 is supplied to the display unit 81 and displayed on the display unit 18.

[0118] This ends the free viewpoint image generation process.

[0119] In the above-mentioned free viewpoint image generation process, the processes of steps S2 and S3 for generating a 3D model of a still subject do not affect the real-time situation at the specified time and place specified by user 91, and therefore can be executed in advance at a timing separate from the free viewpoint image generation process of FIG. 3.

[0120] In the above-mentioned free viewpoint image generation process, if the free viewpoint image generation unit 61 can obtain multiple real-time satellite images corresponding to the specified point and specified time specified by the user 91 from the satellite management device 11, the free viewpoint image generation unit 61 generates a 3D model of the dynamic subject using the obtained multiple real-time satellite images. The condition for obtaining multiple real-time satellite images corresponding to the specified point and specified time is limited to a certain time when the satellite 21 passes above the specified point, and in the most preferable condition, if the satellite 21 is a low-orbit satellite, it will be close to the specified time by about 10 minutes, but if the range that can be estimated from the captured satellite image according to the time error is included, it will be about one hour including 30 minutes before and after the specified time.

[0121] On the other hand, in a time period exceeding a time range of about one hour for the specified location and the specified time and having an error range of several hours, multiple real-time satellite images cannot be acquired, and a real-time image of the dynamic subject is generated based on multiple quasi-real-time satellite images. In this case, there is a first estimation method in which a 3D model of a tentative dynamic subject is estimated to generate a real-time image of the dynamic subject, and a second estimation method in which a quasi-real-time image of a tentative dynamic subject is generated to estimate a real-time image of the dynamic subject. More specifically, in the first estimation method, the free viewpoint image generating unit 61 generates a 3D model of the dynamic subject from multiple quasi-real-time satellite images within an error range of several hours from the specified time, and generates a real-time image of the dynamic subject by estimating a 3D model of the dynamic subject at the specified time using external information. In the second estimation method, the free viewpoint image generation unit 61 generates a 3D model of a hypothetical dynamic subject from multiple quasi-real-time satellite images within an error range of several hours from a specified time, generates a quasi-real-time image of the dynamic subject from a virtual viewpoint specified by the user 91 using the 3D model of the hypothetical dynamic subject, and estimates and generates a real-time image of the dynamic subject from the quasi-real-time image of the dynamic subject using external information.

[0122] Furthermore, in a time period when even quasi-real-time satellite images cannot be obtained, a real-time image of the dynamic subject is generated using real-time external information obtained from the external information providing server 102 and known information about the dynamic subject. For example, a real-time image of the dynamic subject is generated based on position information (AIS information) of an airplane obtained in real time and texture information generated from a known 3D model of the airplane. In order to generate a 3D model of the dynamic subject, multiple real-time or quasi-real-time satellite images are required, but if a two-dimensional satellite image is used as the real-time external information, one satellite image will suffice.

[0123] 3, instead of dividing the cases into three categories, that is, when a real-time satellite image can be acquired (steps S6 to S8), when a quasi-real-time satellite image can be acquired (steps S10 to S13), and when neither a real-time satellite image nor a quasi-real-time satellite image can be acquired (steps S14 and S15), a dynamic subject may be identified and a real-time image may be generated by appropriately combining obtainable real-time satellite images, quasi-real-time satellite images, or external information. As described above, for example, by using a real-time satellite image and a quasi-real-time satellite image in combination, it is possible to obtain an effect such as allowing a real-time satellite image with a low resolution and a wide imaging range to be accepted.

[0124] The real-time satellite image corresponding to a specified location and a specified time, the quasi-real-time satellite image, the real-time external information obtained from the external information providing server 102, and the known information about the dynamic subject, which are used to generate a real-time image of the dynamic subject, can be said to be dynamic subject identification information that identifies the dynamic subject.

[0125] The free viewpoint image generating unit 61 generates a free viewpoint image of a predetermined point on the ground viewed from a predetermined virtual viewpoint in the sky, using a 3D model of a still subject generated using a satellite image captured by the satellite 21 and dynamic subject identification information that identifies a dynamic subject. This allows any point on the ground to be observed from a free viewpoint (virtual viewpoint) in the sky at any time.

[0126] The situation in which there are few satellites 21 capable of imaging at the specified location and time specified by the user 91 is expected to be resolved by an increase in the number of satellites in operation and advances in inter-satellite communication technology.

[0127] In addition, when the user 91 reserves a certain point and time in the future and desires a satellite image seen from a certain virtual viewpoint, the free viewpoint image generating device 51 requests the satellite management device 11 to operate so that the satellite 21 passes at the specified point and time, and the satellite management device 11 can control the satellite management device 11 to pre-position the satellite 21 to be imaged and to instruct the satellite 21 to move its orbit in advance. This allows the free viewpoint image generating device 51 to obtain a satellite image at a specified point and time, and the user 91 to observe the free viewpoint image. The free viewpoint image generating device 51 may obtain orbit information of the satellite 21 from the satellite management device 11 or the like, and present the points and times at which the free viewpoint image can be generated to the user 91, and the user 91 may select a point and time from the list and observe the free viewpoint image.

[0128] A part of the processing executed by the free viewpoint image generating device 51 can be performed by the satellite 21.

[0129] For example, the satellite 21 has a real-time image generating function by the real-time image generating unit 72, generates a real-time image seen from a predetermined virtual viewpoint at a specified location and time specified by the user 91, and downlinks it. When extracting a dynamic subject on the ground by subtracting a real-time image from a base image of the same viewpoint, it is necessary to store the base image in the satellite 21. The free viewpoint image generating device 51 acquires a real-time image generated by the satellite 21 via the satellite management device 11, and generates a free viewpoint image by synthesizing it with a base image generated by itself. In this case, the amount of data required for generating a real-time image to be transmitted from the satellite 21 to the ground station 15 can be reduced. The generation of the real-time image may be performed by the free viewpoint image generating device 51 on the ground by downlinking a 3D model of the dynamic subject from the satellite 21 to the ground station 15. In this case, the amount of downlink data required for generating a real-time image can also be reduced.

[0130] <5. Application Examples> The above-described satellite image processing system 1 can be applied to the following applications. Free viewpoint confirmation of landmarks By generating free viewpoint images of a user's travel destination or a situation where they are flying over a landmark (e.g., Mt. Fuji), they can enjoy the view from space. - Wide area confirmation for disaster areas, etc. By generating free viewpoint images of disaster areas or conflict areas, the situation in those areas can be confirmed. - Three-dimensional satellite images By presenting satellite images for satellite remote sensing, which observe the condition of a target area or object or detect changes in the condition, as free viewpoint images, the images can be viewed as natural, three-dimensional images even from a macro perspective. ·Altitude meteorological observation Free viewpoint images allow for more sophisticated analysis of meteorological phenomena such as clouds by viewing them in 3D from any viewpoint. The results of the analysis can also be used for advanced weather forecasting. -Understanding the growth status of crops and trees The growing conditions of crops and trees can be displayed as 3D visual data using free viewpoint images. The free viewpoint allows for more multifaceted analysis. Urban planning The city situation can be observed three-dimensionally using free viewpoint images. In particular, real-time changes can be observed. At this time, points of change in the city can be visualized by highlighting the changes compared to the base image. Real-time information (vehicle movement status and routes) in the vicinity of a specific point in time can be grasped from multiple viewpoints. - Grasp the situation on construction sites The construction status of facilities in remote locations can be grasped in three dimensions using free viewpoint images. Facilities include buildings such as houses and buildings, dams, oil refineries, factories, ports, etc. In addition, land development status such as land leveling can also be grasped in three dimensions using free viewpoint images. At this time, points of change in the city can be visualized by highlighting the changes compared to the base image. Real-time information (movement status and routes of vehicles, etc.) in the vicinity of a specific point in time can be grasped from multiple viewpoints.

[0131] In addition, an application that links a free viewpoint image that can be generated by the satellite image processing system 1 with an image captured by another imaging device is also conceivable. For example, a system that can switch an image from a free viewpoint image generated by the free viewpoint image generating device 51 to an image captured from the sky at a close distance is conceivable. When a user checks a free viewpoint image as a macro image of a target location and performs a zoom operation on a part of the free viewpoint image, the user can simultaneously check a detailed image of a part of the free viewpoint image by linking the free viewpoint image with an image captured from the sky at a close distance, just as switching to an image captured by a city surveillance camera or a drone.

[0132] <6. Computer configuration example> The series of processes executed by the free viewpoint image generating device 51 can be executed by hardware or software. When the series of processes are executed by software, a program constituting the software is installed in a computer. Here, the computer includes a microcomputer incorporated in dedicated hardware, and a general-purpose personal computer, for example, capable of executing various functions by installing various programs.

[0133] FIG. 4 is a block diagram showing an example of the hardware configuration of a computer that executes a series of processes executed by the free viewpoint image generating device 51 according to a program.

[0134] In the computer, a central processing unit (CPU) 301, a read only memory (ROM) 302, and a random access memory (RAM) 303 are interconnected via a bus 304.

[0135] An input / output interface 305 is further connected to the bus 304. An input unit 306, an output unit 307, a storage unit 308, a communication unit 309, and a drive 310 are connected to the input / output interface 305.

[0136] The input unit 306 includes a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 307 includes a display, a speaker, an output terminal, etc. The storage unit 308 includes a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 309 includes a network interface, etc. The drive 310 drives a removable recording medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0137] In the computer configured as above, the series of processes described above are performed by the CPU 301, for example, loading a program stored in the storage unit 308 into the RAM 303 via the input / output interface 305 and the bus 304 and executing the program. The RAM 303 also stores data and the like necessary for the CPU 301 to execute various processes as appropriate.

[0138] The program executed by the computer (CPU 301) can be provided by being recorded on a removable recording medium 311 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0139] In the computer, the program can be installed in the storage unit 308 via the input / output interface 305 by mounting the removable recording medium 311 in the drive 310. The program can also be received by the communication unit 309 via a wired or wireless transmission medium and installed in the storage unit 308. Alternatively, the program can be installed in the ROM 302 or the storage unit 308 in advance.

[0140] In this specification, the steps described in the flowcharts may be performed chronologically in the order described, but they do not necessarily have to be processed chronologically and may be performed in parallel or at the required timing, such as when a call is made.

[0141] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same case. Therefore, multiple devices housed in separate cases and connected via a network, and a single device in which multiple modules are housed in a single case, are both systems.

[0142] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0143] For example, it is possible to adopt a configuration in which all or part of the above-described multiple embodiments are combined.

[0144] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0145] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0146] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0147] It should be noted that the effects described in this specification are merely examples and are not limiting, and there may be effects other than those described in this specification.

[0148] The present technology can have the following configurations. (1) The system includes an image generating unit that generates a free viewpoint image of a predetermined point on the ground from a predetermined virtual viewpoint in the sky by using a 3D model of a still subject generated by using satellite images captured by an artificial satellite and dynamic subject identification information that identifies a dynamic subject. Image generating device. (2) The image generating unit generates a base image of the 3D model of the still subject viewed from a predetermined virtual viewpoint in the sky as a background image, generates a real-time image viewed from the predetermined virtual viewpoint in the sky as a foreground image using the dynamic subject identification information, and generates the free viewpoint image by synthesizing the base image and the real-time image. The image generating device according to (1) above. (3) The image generating unit uses a 3D model of the dynamic subject as the dynamic subject identification information. The image generating device according to (2) above. (4) The image generating unit generates a real-time image of the 3D model of the dynamic subject viewed from a predetermined virtual viewpoint in the sky as the foreground image. The image generating device according to (3) above. (5) The image generating unit generates the 3D model of the dynamic subject using a real-time satellite image captured by an artificial satellite around a time designated by a user. The image generating device according to (3) or (4). (6) The image generation unit extracts the dynamic subject from the real-time satellite image, and generates a 3D model of the dynamic subject from the extracted image. The image generating device according to (5) above. (7) The image generating unit generates a 3D model of a tentative moving subject using a near real-time satellite image captured by an artificial satellite at a time that is several hours different from a time specified by a user, estimates a 3D model of the dynamic subject at the time specified by the user from the 3D model of the tentative moving subject using external information, and generates the real-time image. The image generating device according to (3) above. (8) The image generating unit generates a 3D model of a tentative dynamic subject using a near real-time satellite image captured by an artificial satellite at a time that differs by several hours from a time specified by a user, generates a near real-time image of the dynamic subject by viewing the 3D model of the tentative dynamic subject from a predetermined virtual viewpoint in the sky, and generates the real-time image from the near real-time image of the dynamic subject using external information. The image generating device according to (3) above. (9) The image generation unit extracts the dynamic subject from the near real-time satellite image, and generates a 3D model of the tentative dynamic subject from the extracted image. The image generating device according to (7) or (8). (10) The image generating unit uses real-time external information and known information about the dynamic subject as the dynamic subject identification information. The image generating device according to (2) above. (11) The image generating unit generates a 3D model of the still subject by using an image from which a moving subject included in a satellite image captured by an artificial satellite has been removed. The image generating device according to any one of (1) to (10). (12) An image generating device, A free viewpoint image is generated by using a 3D model of a stationary subject generated using satellite images captured by an artificial satellite and dynamic subject identification information that identifies a dynamic subject, in which a specific point on the ground is viewed from a specific virtual viewpoint in the sky. Image generation method. (13) Computer, An image generating unit that generates a free viewpoint image of a predetermined point on the ground from a predetermined virtual viewpoint in the sky, using a 3D model of a still subject generated using a satellite image captured by an artificial satellite and dynamic subject identification information that identifies a dynamic subject. A program to function as a [Explanation of symbols]

[0149] 1 satellite image processing system, 11 satellite management device, 21 satellite, 51 free viewpoint image generation device, 61 free viewpoint image generation unit, 62 free viewpoint image storage unit, 63 encoding unit, 65 user IF unit, 66 control unit, 71 base image generation unit, 72 real-time image generation unit, 73 external information acquisition unit, 74 image synthesis unit, 101 satellite image storage server, 102 external information providing server, 301 CPU, 302 ROM, 303 RAM, 306 input unit, 307 output unit, 308 memory unit, 309 communication unit, 310 drive

Claims

1. an image generating unit that generates a free viewpoint image of a predetermined point on the ground from a predetermined virtual viewpoint in the sky by using a 3D model of a still subject generated by using a satellite image captured by an artificial satellite and dynamic subject identification information that identifies a dynamic subject, The image generating unit generates a base image of the 3D model of the still subject viewed from a predetermined virtual viewpoint in the sky as a background image, generates a real-time image viewed from the predetermined virtual viewpoint in the sky as a foreground image using real-time external information and known information about the dynamic subject as the dynamic subject identification information, and generates the free viewpoint image by synthesizing the base image and the real-time image. Image generating device.

2. An image generation unit that generates a free viewpoint image of a predetermined point on the ground from a predetermined virtual viewpoint in the sky using a 3D model of a still subject generated using a satellite image captured by an artificial satellite and dynamic subject identification information that identifies a dynamic subject, the image generation unit generates a base image of the 3D model of the still subject as a background image viewed from a predetermined virtual viewpoint in the sky, generates a real-time image viewed from the predetermined virtual viewpoint in the sky as a foreground image using a 3D model of a hypothetical dynamic subject as the dynamic subject identification information, and generates the free viewpoint image by synthesizing the base image and the real-time image; The image generating unit generates a 3D model of the tentative moving subject using a near real-time satellite image captured by an artificial satellite at a time different by several hours from a time specified by a user, and estimates a 3D model of the moving subject at the time specified by the user from the 3D model of the tentative moving subject using external information, thereby generating the real-time image. Image generating device.

3. An image generation unit that generates a free viewpoint image of a predetermined point on the ground from a predetermined virtual viewpoint in the sky using a 3D model of a still subject generated using a satellite image captured by an artificial satellite and dynamic subject identification information that identifies a dynamic subject, the image generation unit generates a base image of the 3D model of the still subject as a background image viewed from a predetermined virtual viewpoint in the sky, generates a real-time image viewed from the predetermined virtual viewpoint in the sky as a foreground image using a 3D model of a hypothetical dynamic subject as the dynamic subject identification information, and generates the free viewpoint image by synthesizing the base image and the real-time image; The image generating unit generates a 3D model of the tentative dynamic subject using a near real-time satellite image captured by an artificial satellite at a time that differs by several hours from a time specified by a user, generates a near real-time image of the dynamic subject by viewing the 3D model of the tentative dynamic subject from a predetermined virtual viewpoint in the sky, and generates the real-time image from the near real-time image of the dynamic subject using external information. Image generating device.

4. The image generation unit extracts the dynamic subject from the near real-time satellite image, and generates a 3D model of the tentative dynamic subject from the extracted image.

4. An image generating device according to claim 2 or 3.

5. When the image generating unit is able to acquire a plurality of real-time satellite images captured by an artificial satellite near a time designated by a user, the image generating unit generates a 3D model of a dynamic subject using the plurality of real-time satellite images, generates a real-time image of the generated 3D model of the dynamic subject viewed from a predetermined virtual viewpoint in the sky as the foreground image, and generates the free viewpoint image by synthesizing the base image and the real-time image.

5. An image generating device according to claim 1.

6. The image generation unit extracts the dynamic subject from the real-time satellite image, and generates a 3D model of the dynamic subject from the extracted image.

6. The image generating device of claim 5.

7. The image generating unit generates a 3D model of the still subject by using an image from which a moving subject included in a satellite image captured by an artificial satellite has been removed.

7. An image generating device according to claim 1.

8. An image generating device, generating a free viewpoint image of a predetermined point on the ground viewed from a predetermined virtual viewpoint in the sky using a 3D model of a still subject generated using a satellite image captured by an artificial satellite and dynamic subject identification information that identifies a dynamic subject; a base image of the 3D model of the still subject viewed from a predetermined virtual viewpoint in the sky is generated as a background image, and a real-time image viewed from the predetermined virtual viewpoint in the sky is generated as a foreground image using real-time external information and known information about the dynamic subject as the dynamic subject identification information, and the free viewpoint image is generated by synthesizing the base image and the real-time image; An image generating method comprising:

9. Computer, An image generating unit that generates a free viewpoint image of a predetermined point on the ground from a predetermined virtual viewpoint in the sky by using a 3D model of a still subject generated by using a satellite image captured by an artificial satellite and dynamic subject identification information that identifies a dynamic subject. Functioning as a The image generating unit generates a base image of the 3D model of the still subject viewed from a predetermined virtual viewpoint in the sky as a background image, generates a real-time image viewed from the predetermined virtual viewpoint in the sky as a foreground image using real-time external information and known information about the dynamic subject as the dynamic subject identification information, and generates the free viewpoint image by synthesizing the base image and the real-time image. Program for.

Images