Image generation device, image generation method, and program product

Abstract

The present technology relates to an image generation device, an image generation method, and a program that enable observation of an image of an arbitrary place on the Earth from a free viewpoint in the sky at an arbitrary time. The image generation device is provided with an image generation unit that uses a 3D model of a still subject generated by using a satellite image captured by an artificial satellite and dynamic subject specification information that specifies a dynamic subject, and generates a free viewpoint image obtained by imaging a specified place on the Earth from a specified virtual viewpoint in the sky. The present technology can be applied to an image generation device that generates a free viewpoint image from a satellite image captured by an artificial satellite.

Description

Technical Field

[0001] This technology relates to image generation devices, image generation methods and programs, and more particularly to image generation devices, image generation methods and programs capable of observing any location on the ground from a free viewpoint in the sky at any time. Background Technology

[0002] Earth observation is conducted by observation satellites equipped with imaging devices (see, for example, PTL 1). In particular, the number of small low-Earth orbit observation satellites has increased in recent years.

[0003] In the field of virtual reality (VR) technology, a 3D model is used that allows an object to be reproduced and displayed as a subject from any free viewpoint. The 3D data of the subject's 3D model is converted into, for example, multiple texture images and depth images captured from multiple viewpoints, sent to a reproduction device, and displayed on the reproduction side (e.g., refer to PTL 2).

[0004] Reference List

[0005] Patent documents

[0006] PTL 1: JP 2006-277007 A

[0007] PTL 2:WO 2017 / 082076 Summary of the Invention

[0008] Technical issues

[0009] When observing Earth, one might desire to observe any location on the ground from a free viewpoint in the sky at any time, much like in VR. However, with geostationary satellites, it's difficult to observe at high resolution due to their altitude of 36,000 kilometers, and because their position is fixed, only specific locations can be observed. Furthermore, with low-Earth orbit or medium-Earth orbit satellites, observation is only possible from an orbital viewpoint, not a free viewpoint. Therefore, it is currently impossible to fully satisfy the desire to observe any location on the ground from a free viewpoint in the sky at any time.

[0010] Given this situation, this technology enables the observation of any location on the ground from a free viewpoint in the sky at any time.

[0011] Solution to the problem

[0012] One aspect of this technology is an image generation device that includes an image generation unit that uses a 3D model of a stationary subject generated from satellite images captured by an artificial satellite and dynamic subject identification information that identifies a dynamic subject to generate a free-viewpoint image of a predetermined location on the ground viewed from a predetermined virtual viewpoint in the sky.

[0013] One aspect of this technology, an image generation method, includes generating a free-viewpoint image of a predetermined location on the ground from a predetermined virtual viewpoint in the sky, using a 3D model of a stationary subject generated from satellite images captured by an artificial satellite and dynamic subject identification information that identifies a dynamic subject, by an image generation device.

[0014] One aspect of this technology involves a computer used as an image generation unit that uses a 3D model of a stationary subject generated from satellite images captured by an artificial satellite and dynamic subject identification information to generate a free-viewpoint image of a predetermined location on the ground viewed from a predetermined virtual viewpoint in the sky.

[0015] In one aspect of this technology, a free-viewpoint image of a predetermined location on the ground is generated by using a 3D model of a stationary subject generated from satellite images captured by an artificial satellite and dynamic subject identification information that identifies a moving subject.

[0016] One aspect of this technology is an image generation device that can be implemented by causing a computer to execute a program. The program executed by the computer can be provided by transmission via a transmission medium or by recording on a recording medium.

[0017] Image generation devices can be standalone devices or internal blocks that constitute a device. Attached Figure Description

[0018] Figure 1 This is a block diagram illustrating a configuration example of a satellite image processing system that applies the present technology.

[0019] Figure 2 This is a block diagram illustrating a detailed configuration example of a free-viewpoint image generation device.

[0020] Figure 3 This is a flowchart illustrating the free-viewpoint image generation process performed by a free-viewpoint image generation device.

[0021] Figure 4 This is a block diagram illustrating a configuration example of a computer to which the present technology is applied. Detailed Implementation

[0022] Hereinafter, embodiments for implementing the present technology (hereinafter referred to as embodiments) will be described with reference to the accompanying drawings. Furthermore, in this specification and the drawings, components having substantially the same functional configuration will be indicated by the same reference numerals, and therefore repeated descriptions thereof will be omitted. The descriptions will proceed in the following order.

[0023] 1. Configuration example of a satellite image processing system

[0024] 2. Volume capture technology

[0025] 3. Configuration example of a free-viewpoint image generation device

[0026] 4. Free viewpoint image generation and processing

[0027] 5. Application Examples

[0028] 6. Computer configuration example

[0029] <1. Configuration Example of Satellite Image Processing System>

[0030] Figure 1 This is a block diagram illustrating a configuration example of a satellite image processing system that applies the present technology.

[0031] Figure 1 The satellite image processing system 1 is a system capable of observing any location on the ground from a free viewpoint in the sky at any time using images captured by multiple artificial satellites (hereinafter referred to as satellites) (hereinafter referred to as satellite images). In this embodiment, the satellite is equipped with imaging equipment and has at least the function of imaging the ground.

[0032] The satellite operating company has satellite management equipment 11 for managing multiple satellites 21 and multiple communication devices 13 for communicating with the satellites 21. Some of the satellite management equipment 11 and the multiple communication devices 13 may be owned by companies other than the satellite operating company. The satellite management equipment 11 is connected to the multiple communication devices 13 via a predetermined network 12. The communication devices 13 are located in a ground station (ground base station) 15. Although Figure 1 An example is shown where the number of communication devices 13 is three (i.e., communication devices 13A to 13C), but the number of communication devices 13 is arbitrary.

[0033] Satellite management equipment 11 manages multiple satellites 21 owned by a satellite operating company. Specifically, satellite management equipment 11 obtains relevant information from information providing servers 41 of one or more external organizations as needed and determines the operation plans for the multiple satellites 21 owned by them. Then, in response to customer expectations, satellite management equipment 11 instructs designated satellites 21 to capture images via corresponding communication devices 13. Furthermore, satellite management equipment 11 acquires and stores satellite images transmitted from satellites 21 via communication devices 13. The acquired satellite images undergo predetermined image processing as needed and are provided (transmitted) to the customer. Additionally, the acquired satellite images are provided (transmitted) to a free-viewpoint image generation device 51 of an image providing company.

[0034] An information providing server 41, installed in an external organization, responds to requests from satellite management equipment 11 or periodically provides predetermined relevant information to satellite management equipment 11 via a predetermined network. For example, the relevant information provided by the information providing server 41 includes, for instance, satellite orbit information in Two Line Elements (TLE) format, which can be obtained from the North American Aerospace Defense Command (NORAD), an external organization. Additionally, meteorological information, such as weather and cloud cover at predetermined locations on the ground, can be obtained from a meteorological information providing company, an external organization.

[0035] The free-viewpoint image generation device 51 performs image processing to generate a free-viewpoint image using satellite images captured by satellite 21 provided via a predetermined network from satellite management device 11. This free-viewpoint image is a satellite image observed from a free-viewpoint in the sky at any time, targeting any location on the ground. The free-viewpoint image generation device 51 responds to input from user 91 ( Figure 2 The satellite management device 11 sends a request to the satellite management device 11 to capture satellite images as needed. The free-viewpoint image generation process can be performed by the satellite operating company, and in this case, the satellite operating company and the image providing company are the same company. Furthermore, the satellite management device 11 and the free-viewpoint image generation device 51 can be implemented as a single device.

[0036] Under the control of satellite management equipment 11, communication equipment 13 communicates with a predetermined satellite 21 designated by satellite management equipment 11 via antenna 14. For example, communication equipment 13 sends imaging instructions to the predetermined satellite 21 at a predetermined time and location for imaging a predetermined area on the ground. Furthermore, communication equipment 13 receives satellite images transmitted from satellite 21 and provides them to satellite management equipment 11 via network 12. The transmission from communication equipment 13 at ground station 15 to satellite 21 is also referred to as an uplink, and the transmission from satellite 21 to communication equipment 13 is also referred to as a downlink. Communication equipment 13 can perform direct communication with satellite 21 and can also perform communication via relay satellite 22. For example, a geostationary satellite is used as the relay satellite 22.

[0037] Network 12 and the network between information providing server 41 or free-viewpoint image generation device 51 and satellite management device 11 are any communication networks, which can be wired communication networks, wireless communication networks, or both. Furthermore, network 12 and the network between information providing server 41 or free-viewpoint image generation device 51 and satellite management device 11 can be configured as a single communication network or multiple communication networks. For example, these networks can be configured as communication networks or communication channels of any communication standard, such as the Internet, public telephone network, wide-area wireless mobile communication network (e.g., so-called 4G or 5G networks), wireless communication networks performing communication conforming to wide area network (WAN), local area network (LAN), or Bluetooth (registered trademark) standards, communication channels for short-range wireless communication such as near field communication (NFC), infrared communication channels, and wired communication networks conforming to standards such as High Definition Multimedia Interface (HDMI) (registered trademark) or Universal Serial Bus (USB).

[0038] Each of the multiple satellites 21 can operate as a single satellite, or as a group of multiple satellites. Figure 1 In the example, the illustration of satellite 21 operating as a single satellite is omitted, as are satellites 21A and 21B forming the first satellite group 31A, and satellites 21C and 21D forming the second satellite group 31B. Although for simplicity, in Figure 1 The example shown illustrates a satellite group 31 consisting of two satellites 21, but the number of satellites 21 constituting a satellite group 31 is not limited to two.

[0039] As a system that operates multiple satellites 21 as a unit (satellite group), there are constellations and formation flying. A constellation is a system that provides services primarily and uniformly worldwide by launching a large number of satellites into one or more orbital planes. Each satellite constellation has a predetermined function, and multiple satellites are operated for purposes such as increasing observation frequency. On the other hand, formation flying is a system in which multiple satellites are deployed over a narrow area of ​​approximately several kilometers while maintaining relative positional relationships. Formation flying can provide services that a single satellite cannot provide, such as high-precision 3D measurement and velocity detection of moving objects. In this embodiment, the satellite group is operated through either constellation or formation flying.

[0040] When communication device 13 communicates with each satellite 21 constituting satellite group 31, there exists... Figure 1 The first satellite group 31A communicates with each satellite 21 in the same manner as the second satellite group 31B, but only one satellite 21C (hereinafter also referred to as representative satellite 21C) communicates with the communication device 13, while the other satellites 21D communicate indirectly with the communication device 13 through inter-satellite communication with representative satellite 21C. The method of communication with the ground station 15 (communication device 13) can be predetermined by the satellite group 31, or appropriately selected based on the details of the communication. Satellite 21 operating as a single satellite can also communicate with the communication device 13 of the ground station 15, or via relay satellite 22. When satellite 21 moves within the communication range of the communication device 13 installed in the ground station 15, satellite 21 can downlink to the communication device 13 at a predetermined frequency. When satellite 21 moves outside the communication range of communication equipment 13 installed in ground station 15, satellite 21 can transmit to other satellites 21 located within the communication range of communication equipment 13 installed in ground station 15 according to inter-satellite communication, and can also transmit downlink to ground station 15 via other satellites 21. Therefore, the real-time characteristics of satellite images can be guaranteed.

[0041] The satellite image processing system 1 is configured as described above.

[0042] The satellite management device 11 or the free-viewpoint image generation device 51 can perform the following image processing on the satellite images captured by each satellite 21.

[0043] (1) Generation of metadata

[0044] Metadata can be generated based on information transmitted from satellite 21 and information about satellite 21 that has performed imaging. For example, information such as the latitude and longitude of the imaged object's location, and information about satellite 21's attitude control and acceleration during imaging can be generated as metadata. Metadata can be generated by satellite 21 based on conditions during imaging, and in this case, metadata pre-added to satellite images captured by satellite 21 can be used.

[0045] (2) Satellite image correction processing

[0046] It can perform correction processes, such as radiometric correction of sensitivity characteristics, geometric correction of the orbital position and attitude error of satellite 21, orthophoto correction for correcting geometric distortion caused by terrain height differences, and map projection for projection onto a map projection surface.

[0047] (3) Color synthesis processing

[0048] It can perform color compositing processes, such as pan sharpening processing, true color compositing, pseudo color compositing, natural color compositing, SAR image compositing, and processing for adding color to satellite images for each band.

[0049] (4) Other image synthesis

[0050] It can perform the synthesis of satellite images captured by satellite (satellite 21) in the past, satellite images captured by other satellites 21 and some other images, the synthesis of satellite images captured in different bands, the synthesis of map information, etc.

[0051] (5) Information extraction

[0052] Vegetation detection information (e.g., Normalized Differential Vegetation Index (NDVI)) and water body detection information (e.g., Normalized Differential Water Index (NDWI)) can be calculated using different bands such as R (red) and IR (infrared). Features such as highlighting of specific subjects (e.g., vehicles, moving objects, or schools of fish), extracting information about specific bands, and identifying points of change relative to previous images can be performed.

[0053] Specifically, when using multiple satellite images captured by multiple satellites 21 flying in constellations or formations, the satellite management device 11 or the free-viewpoint image generation device 51 can perform image processing as described below more efficiently.

[0054] (1) High resolution or high quality processing

[0055] By overlaying multiple satellite images, satellite images with improved resolution can be generated. Furthermore, by synthesizing panchromatic sharpened images obtained by combining monochrome and color images, as well as satellite images with different imaging conditions (e.g., different dynamic ranges and shutter speeds, different bands (wavelength bands), and different resolutions), satellite images with improved resolution can be generated.

[0056] (2) Function sharing

[0057] Different bands, such as R (red) and IR (infrared), can be used to calculate indices such as the Normalized Differential Vegetation Index (NDVI).

[0058] (3) 3D measurement

[0059] Three-dimensional information can be obtained from parallax images. Furthermore, by using three-dimensional information, the accuracy of object recognition on the ground can be improved. For example, it can be determined whether an object is a vehicle (even if the image does not immediately indicate that the object is a vehicle due to resolution, it can be inferred that the object on the road is a three-dimensional object rather than a pattern).

[0060] (4) Differential measurement

[0061] Multiple satellite images captured from the same location at different times can be used to extract changes between a first time and a second time. Furthermore, imaging can be performed that extracts and colors only the changed objects. Additionally, for example, multiple satellite images can be used to calculate the speed of a ship or vehicle, or to calculate wind speed based on the movement of clouds, etc.

[0062] (5) Other image synthesis

[0063] It can also perform the synthesis of past satellite images and satellite images captured by other satellites 21, the synthesis of satellite images captured in different bands, the synthesis of map information, etc.

[0064] Each satellite 21 can send satellite images obtained by imaging the ground as raw data to the communication device 13, or it can send satellite images after performing the above image processing.

[0065] (Image format)

[0066] The processed images and satellite images are stored in the storage unit of each device and sent to other devices using, for example, the following image formats.

[0067] (1) CEOS

[0068] CEOS is a format standardized by the Committee on Earth Observation Satellites. CEOS includes "CEOS-BSQ" which divides the file for each band and "CEOS-BIL" which multiplexes multiple bands.

[0069] (2) HDF

[0070] It is a format developed by the National Center for Supercomputing Applications (NCSA) at the University of Illinois. Multiple bands are combined into a single file, making data easily exchangeable in various computing environments.

[0071] (3) Geographic TIFF

[0072] This is a format for adding information used in remote sensing to a tagged image file format (TIFF). Because it's in TIFF format, it can be opened using general image viewers.

[0073] (4) JPEG2000

[0074] This is an image format standardized by the Joint Group of Image Experts. JPEG2000 not only improves the compression ratio, but also employs techniques for improving the image of regions of interest and copyright protection techniques (such as digital watermarking).

[0075] <2. Volume Capture Technology>

[0076] A technique is known for providing free-viewpoint images by generating a 3D model of the subject from images captured from multiple viewpoints (including moving images) and a virtual viewpoint image of the 3D model based on an arbitrary viewing position. This technique is also known as volumetric capture.

[0077] The free-viewpoint image generation device 51 applies volumetric capture technology to satellite images to generate free-viewpoint images, which are satellite images observed from a free viewpoint in the sky at any time for any location on the ground.

[0078] Therefore, firstly, we will briefly describe the generation of a 3D model of the subject when people are the subject, and the display of a free-viewpoint image based on the generated 3D model.

[0079] When generating a 3D model, multiple captured images can be obtained by using multiple imaging devices to image the predetermined imaging space from its vicinity outside the predetermined imaging space, wherein the subject is arranged in the predetermined imaging space. The captured images may consist of, for example, moving images.

[0080] The subject in the foreground, which is a display object in the imaging space, is extracted using captured images obtained from multiple imaging devices in different directions, and a 3D object (3D modeling) is generated as a 3D model of the subject. For example, the 3D object is generated by means of a visual hull, a multi-view stereo, etc., which projects the outline of the subject at each viewpoint into 3D space and makes the intersection area of ​​the outline a 3D shape, and the multi-view stereo uses the consistency of texture information between viewpoints.

[0081] Then, among one or more 3D objects existing in the imaging space, the data of one or more 3D objects (hereinafter also referred to as 3D model data) are sent to the playback device and reproduced. That is, the playback device renders the 3D objects based on the acquired 3D object data, so that the 3D shape video is displayed on the viewer's viewing device. The viewing device consists of, for example, a liquid crystal display, a head-mounted display, etc.

[0082] Various formats can be used as the data format for 3D model data.

[0083] One data format is one where the geometric information (shape information) of an object is represented by a set of points (point cloud) at the object's three-dimensional location, and the object's color information is preserved for each point. In this format, one set of geometric and color information is preserved for each object. This format is described as a point cloud format.

[0084] Another data format is one where the geometric information of an object is represented by a set of points (point cloud), as in the aforementioned point cloud format, or by vertices of a polygonal mesh and the connections between those vertices, and the object's color information is preserved in the captured images (two-dimensional texture images) captured by each imaging device. In this format, one set of geometric information and color information consisting of the same number of captured images (two-dimensional texture images) as the number of imaging devices are preserved for each object. This format is called a multi-texture geometry format.

[0085] Another data format is one in which the geometry of an object is represented by a polygonal mesh, and the object's color information is maintained corresponding to each polygonal mesh. A two-dimensional texture image, as the color information attached to each polygonal mesh, is represented by a UV coordinate system. In this format, for an object, one set of geometric information and color information consisting of a two-dimensional texture image are maintained. In this embodiment, this format is described as the UV texture geometry format. The UV texture geometry format is a format standardized by the MPEG-4 Animation Framework Extension (AFX).

[0086] Another data format is one in which the geometric information of an object is represented by distance information corresponding to the captured images captured by each imaging device, and the object's color information is preserved in the captured images (two-dimensional texture images) captured by each imaging device. As the distance information corresponding to the captured images captured by each imaging device, a depth image is used, where the distance to the subject in the depth direction is stored as a depth value corresponding to each pixel of the captured image. In this format, for an object, geometric information consisting of the same number of depth images as the number of imaging devices and color information consisting of the same number of captured images (two-dimensional texture images) as the number of imaging devices are preserved. This format is described as a multi-texture depth format. The advantage of the multi-texture depth format is that when transmitting 3D model data, AVC (Advanced Video Coding), HEVC (High-Efficiency Video Coding), etc., can be used as encoding methods as is, thus achieving efficient compression.

[0087] Any of the various data formats mentioned above for 3D model data can be used. The rendering side can specify the data format, or the distribution side can determine the data format. Furthermore, the data format can be pre-determined for each application.

[0088] The rendering side can request only the 3D object to be viewed from among one or more 3D objects existing in the imaging space, and display that 3D object on the viewing device. For example, the rendering side assumes that the viewer's viewing range is a virtual camera within the imaging range, and requests only the 3D object captured by the virtual camera from among a large number of 3D objects existing in the imaging space, and displays that 3D object on the viewing device. The viewpoint (virtual viewpoint) of the virtual camera can be set to an arbitrary position, allowing the viewer to view the subject from any viewpoint in the real world. A background image representing a predetermined space is appropriately combined with the 3D object. The background image can be a still image with a fixed virtual viewpoint, or it can be an image that changes according to the virtual viewpoint, such as the subject serving as a foreground image.

[0089] <3. Configuration Example of Free Viewpoint Image Generation Device>

[0090] Figure 2 This is a block diagram illustrating a detailed configuration example of the free viewpoint image generation device 51.

[0091] The free-viewpoint image generation device 51 includes a free-viewpoint image generation unit 61, a free-viewpoint image accumulation unit 62, an encoding unit 63, a communication unit 64, and a user interface (IF) unit 65. The user IF unit 65 includes a display unit 81 and an operation unit 82.

[0092] The free-viewpoint image generation device 51 can receive operations from user 91 via user IF unit 65, perform processing according to user 91's instructions, and perform processing according to the instructions even when user 91 instructs the free-viewpoint image generation device 51 to perform predetermined processing via terminal device 92. User 91 instructs the generation of a free-viewpoint image by inputting instructions to operation unit 82 directly or via terminal device 92. The free-viewpoint image is a satellite image of a predetermined location (designated location) on the ground viewed from a predetermined virtual viewpoint in the sky at a predetermined time (specified time).

[0093] In response to an instruction from a user to generate a free-viewpoint image, the free-viewpoint image generation device 51 generates a satellite image of a specified location on the ground viewed from a predetermined virtual viewpoint at a specified time by synthesizing a base image viewed from a predetermined virtual viewpoint and a real-time image.

[0094] The base image viewed from a predetermined virtual viewpoint is a satellite image corresponding to the background image of the volumetric capture technique, and is a satellite image of a stationary subject that does not change over a certain period of time (regardless of any changes within that period of time). The base image is an image with relatively few changes compared to the live image.

[0095] On the other hand, the real-time image viewed from the predetermined virtual viewpoint is a satellite image corresponding to the foreground image (subject) of the volumetric capture technology, and is a satellite image of a dynamic subject that changes in real time (real-time subject).

[0096] A subject that changes in real time (real-time subject) is a subject that changes over a period of time, in addition to a stationary subject included in the base image, and includes, for example, subjects that change from an instantaneous moment to subjects that change over several hours to about a day.

[0097] For example, moving subjects include: moving bodies such as airplanes, ships, vehicles and people; meteorological phenomena such as clouds, auroras and volcanic eruptions; the reflection of the sun on the ocean, lakes, rivers and the ground; the sky colors at sunrise and sunset and the light information of shadows, etc.

[0098] This assumes that subjects that change over a period of approximately one day or longer, other than dynamic subjects, are included in static subjects. For example, changes in the color of mountains such as autumn leaves, the state of rice paddies immediately following planting, and the state of rice ears during harvest season can be considered seasonal changes, and therefore forests and rice paddies are included in static subjects.

[0099] The free viewpoint image generation unit 61 includes a basic image generation unit 71, a real-time image generation unit 72, an external information acquisition unit 73, and an image synthesis unit 74.

[0100] The free-viewpoint image generation unit 61 of the free-viewpoint image generation device 51 is connected to the satellite image accumulation server 101 and the external information providing server 102 via a predetermined network. The network between the free-viewpoint image generation device 51 and the satellite image accumulation server 101 or the external information providing server 102 can be any communication network similar to the network 12 described above. The connection to the predetermined network is performed via the communication unit 64.

[0101] The free viewpoint image generation unit 61 generates a free viewpoint image of a predetermined location on the ground from a predetermined virtual viewpoint in the sky, based on the generation instruction from the control unit 66, and provides the free viewpoint image to the free viewpoint image accumulation unit 62.

[0102] The base image generation unit 71 acquires multiple satellite images accumulated (stored) in the satellite image accumulation server 101 and captured from different viewpoints, and uses the acquired multiple satellite images to generate a 3D model of a stationary subject on the ground. Then, the base image generation unit 71 generates a first virtual viewpoint image when viewing the 3D model of the stationary subject on the ground from a predetermined virtual viewpoint specified by the user 91, and provides the first virtual viewpoint image as a base image to the image synthesis unit 74.

[0103] The base image generation unit 71 can be used to generate a 3D model of a stationary subject on the ground after removing moving subjects included in multiple satellite images acquired from the satellite image accumulation server 101. In other words, the base image generation unit 71 generates satellite images obtained by removing subjects to be included in real-time images generated by the real-time image generation unit 72 from satellite images acquired from the satellite image accumulation server 101, and can be used to generate a 3D model of a stationary subject on the ground. For example, moving subjects included in satellite images can be removed by comparing captured satellite images of the same location, detecting mismatched subjects as moving subjects, and removing those subjects. Furthermore, for example, satellite images captured by multiple artificial satellites (e.g., in formation) can be used to detect and remove minute changes that are moving subjects.

[0104] Multiple satellite images obtained by imaging the ground from the sky using artificial satellites equipped with imaging devices are accumulated in a satellite image accumulation server 101. The artificial satellite can be satellite 21 of a satellite operating company that owns satellite management equipment 11, or it can be a satellite of another company. The satellite image accumulation server 101 can be a server operated by the satellite operating company that owns satellite management equipment 11, a server operated by an image providing company that owns free-viewpoint image generation equipment 51, or a server operated by another company. Furthermore, the satellite image accumulation server 101 can be included as a satellite image accumulation unit within the free-viewpoint image generation equipment 51, and accumulate satellite images provided from satellite management equipment 11.

[0105] The satellite images accumulated in the satellite image accumulation server 101 are images obtained by imaging the ground from multiple viewpoints in the sky using one or more artificial satellites. When imaging is performed using a single artificial satellite, multiple satellite images corresponding to multiple viewpoints are generated by imaging the ground in a time-division manner. When multiple artificial satellites are used, each artificial satellite can generate a large number of satellite images corresponding to a large number of viewpoints by imaging the ground in a time-division manner.

[0106] The satellite images accumulated in the satellite image accumulation server 101 can be satellite images obtained by combining two satellite images captured by artificial satellites into one. For example, a satellite image can be obtained by performing a stitching process on two satellite images with partially overlapping imaging areas. The two satellite images to be stitched can be images with different resolutions.

[0107] Instead of satellite images accumulated in satellite image accumulation server 101, aerial images captured by aircraft can be used.

[0108] The satellite images accumulated in the satellite image accumulation server 101 are expected to have high resolution because the precision of subsequent processing changes. For example, a ground resolution of 1m or less is expected, and a resolution of 50cm or less is expected when it is desired to identify ground structures (such as vehicles).

[0109] In addition to satellite imagery as texture information, depth images, which are information about the depth to a stationary subject, can also be accumulated in satellite imagery accumulation server 101. Depth images can be generated, for example, based on parallax information from multiple satellite images obtained by capturing the same location from different viewpoints using one or more satellites. Imaging by multiple satellites includes imaging via formation flying. Furthermore, depth images can be generated based on altitude measurements obtained from synthetic aperture radar (SAR) satellites. Alternatively, depth images can be generated by estimation from 2D satellite images, for example, by estimating altitude information based on the size of shadows reflected in the satellite images, and so on.

[0110] The satellite images accumulated in the satellite image accumulation server 101 can be accumulated for each of the following conditions: different times (e.g., morning, noon, afternoon, evening, and night), different types of weather (e.g., clear, sunny, cloudy, and rainy), and seasons (e.g., spring, summer, autumn, and winter). In this case, the base image generation unit 71 can generate a 3D model of a stationary subject on the ground for each accumulation condition. When satellite images under predetermined conditions have not yet been accumulated in the satellite image accumulation server 101, the base image generation unit 71 can perform estimation based on satellite images of the same location captured under other conditions to generate satellite images that have not yet been accumulated, and use them for the processing of generating a 3D model of a stationary subject. For example, when autumn satellite images have not yet been accumulated, autumn satellite images can be estimated and generated based on summer or winter satellite images of the same location.

[0111] The satellite image accumulation server 101 can accumulate data of 3D models of stationary subjects on the ground, generated through 3D modeling performed by other devices, rather than satellite images captured from numerous viewpoints in the sky. In this case, the basic image generation unit 71 acquires the accumulated 3D model data of stationary subjects on the ground in the satellite image accumulation server 101 and generates a first virtual viewpoint image based on the acquired 3D model data of stationary subjects on the ground. In other words, when accumulating 3D model data of stationary subjects on the ground in the satellite image accumulation server 101, 3D modeling processing is omitted.

[0112] The data format of the 3D model of the stationary subject on the ground accumulated in the satellite image accumulation server 101 can be any of the aforementioned 3D model data formats. When the basic image generation unit 71 generates the 3D model of the stationary subject on the ground, the data format can be any format.

[0113] Since the base image is a satellite image of a stationary subject that does not affect the specified time of the free-viewpoint image specified by user 91, the satellite images accumulated in the satellite image accumulation server 101 can be satellite images captured approximately one week to one month before the specified time of the free-viewpoint image specified by user 91. Alternatively, the satellite images accumulated in the satellite image accumulation server 101 can be satellite images captured (in the future) after the specified time of the free-viewpoint image specified by user 91.

[0114] Satellite images acquired at a newer imaging time or with higher resolution, obtained by capturing the same location, can be provided from the real-time image generation unit 72, instead of satellite images acquired from the satellite image accumulation server 101. When a newer or higher-resolution satellite image is provided from the real-time image generation unit 72, the base image generation unit 71 can use the satellite image to update the 3D model of a stationary subject on the ground and generate a base image.

[0115] The real-time image generation unit 72 acquires multiple satellite images from different viewpoints corresponding to a specified location and time specified by the user 91 from the satellite management device 11. The specified location and time are the location and time at which the free-viewpoint image will be generated. The multiple satellite images from different viewpoints corresponding to the specified time are called real-time satellite images to distinguish them from the satellite images used to generate the base image.

[0116] The real-time image generation unit 72 provides the satellite management device 11 with a specified location and time of a free-viewpoint image, and acquires multiple satellite images (real-time satellite images) obtained by capturing the specified location from different viewpoints at the specified time. The satellite management device 11 sends an imaging instruction from the communication device 13 to the satellite 21 passing the specified location near the specified time provided by the real-time image generation unit 72, and causes the satellite 21 to capture an image. The time near the specified time may include an error ranging from several minutes to tens of minutes. The satellite management device 11 or the real-time image generation unit 72 can modify the captured satellite image to an image that has been estimated to reflect changes corresponding to the time error. For example, if an aircraft flying at a predetermined location at a predetermined time is imaged as a moving subject, and this time has an error relative to the specified time, the image can be modified to a satellite image in which the aircraft's position has been moved according to that time error.

[0117] Multiple satellite images from different viewpoints can be acquired by a single satellite 21 in a time-division manner, or by multiple satellites 21 with different orbits from their respective viewpoints. Alternatively, multiple satellites 21 with the same orbit or multiple satellites 21 operating in formation can acquire satellite images with a time difference of approximately several minutes to tens of minutes. Regarding satellite images with a time difference of several minutes to tens of minutes, other satellite images can be modified to images whose changes have been estimated corresponding to the time error, based on the satellite image closest to the specified time.

[0118] The real-time image generation unit 72 extracts dynamic subjects on the ground from each of the acquired multiple real-time satellite images. For example, dynamic subjects on the ground can be extracted by comparing them with a base image from the same viewpoint. Furthermore, since clouds, aircraft, ships, vehicles, etc., have characteristic shapes and colors, dynamic subjects can be extracted based on image recognition processing based on these features. Alternatively, dynamic subjects can be extracted using the same processing performed by the base image generation unit 71 to remove dynamic subjects.

[0119] The real-time image generation unit 72 generates a 3D model of the dynamic subject based on each real-time satellite image that includes only the extracted dynamic subject on the ground. Then, the real-time image generation unit 72 generates a second virtual viewpoint image as a real-time image of the dynamic subject when the 3D model of the dynamic subject is viewed from a predetermined virtual viewpoint specified by the user 91, and provides the second virtual viewpoint image to the image synthesis unit 74.

[0120] The real-time image generation unit 72 can generate 3D information based on a parallax image calculated from multiple satellite images obtained by capturing a specified location from different viewpoints, rather than generating a 3D model of a dynamic subject. Alternatively, the real-time image generation unit 72 can generate 3D information based on estimations made using only 2D satellite images without calculating depth information, rather than generating a 3D model of a dynamic subject. For example, an aircraft can be extracted as a dynamic subject from a 2D satellite image, and 3D information of the dynamic subject can be generated based on the aircraft's flight altitude, which is known information.

[0121] The data format used by the real-time image generation unit 72 to generate a 3D model of a dynamic subject can be any of the aforementioned 3D model data formats. If no 3D model is generated, only 2D satellite images are stored in the internal memory.

[0122] When real-time satellite images have been acquired from satellite management device 11, real-time image generation unit 72 can provide the acquired real-time satellite images to base image generation unit 71 as updated satellite images for updating satellite images of the same location.

[0123] If a satellite 21 passing by a specified location exists near the specified time provided by the real-time image generation unit 72, then the satellite 21 can perform imaging. However, in reality, a satellite 21 passing by a specified location may not actually exist near the specified time. In such cases, the real-time image generation unit 72 uses external information obtained by the external information acquisition unit 73 from the external information providing server 102 to generate a real-time image of the moving subject, and provides the real-time image to the image synthesis unit 74.

[0124] When the moving subject is a mobile object such as an airplane, ship, vehicle, or person, the location information of the mobile object at a specified time is obtained from an external information providing server 102 based on information about a specified location and time in the free-viewpoint image. For example, regarding ships, airplanes, etc., the location information of the mobile object at a specified time can be obtained by obtaining Automatic Identification System (AIS) information from the external information providing server 102. Regarding vehicles or people, the location information of the mobile object at a specified time can be obtained by obtaining location information from devices installed on the vehicle or person and location information detected by surveillance cameras installed on the ground from the external information providing server 102. Location information of various mobile objects can be obtained from the external information providing server 102 operated by an operating company that provides location information of mobile objects.

[0125] The real-time image generation unit 72 generates a real-time image of the moving subject by acquiring known 3D model or texture information of the moving subject from an external company or generating known 3D model or texture information of the moving subject in its own company, pre-storing known 3D model or texture information of the moving subject internally, and arranging the 2D image of the moving subject at a predetermined position based on the position information of the moving subject acquired by the external information acquisition unit 73, and provides the real-time image to the image synthesis unit 74.

[0126] When the moving subject is a meteorological phenomenon such as cloud distribution, a real-time image of cloud distribution can be generated by obtaining information indicating cloud distribution and estimated altitude information from an external information providing server 102 operated by a meteorological service company that provides meteorological information, and by placing a 2D image of the clouds at the estimated altitude. Furthermore, for example, when the moving subject is a phenomenon of solar reflection, a real-time image of light information such as solar shadows and reflections can be generated by obtaining information such as the sun's position from the external information providing server 102 operated by a meteorological service company, estimating light information, and placing the estimated information at a predetermined location.

[0127] Although for the sake of ease of description, Figure 2Only one external information providing server 102 is shown, but the external information acquisition unit 73 can obtain the desired external information by accessing external information providing servers 102 in different external information providing companies or different locations, depending on the type of external information to be acquired.

[0128] Although the methods for generating real-time images of a moving subject by the real-time image generation unit 72 have been described above, both when satellite images corresponding to the location and time specified by user 91 can be captured and when such satellite images cannot be captured, there may actually be intermediate situations between these cases. That is, there are situations where satellite images of a specified location with an error range of tens of minutes relative to the time specified by user 91 cannot be captured, but satellite images of the specified location from 1 hour ago, 3 hours ago, 6 hours ago, and within an error range of several hours can be acquired.

[0129] In this configuration, the real-time image generation unit 72 acquires multiple satellite images captured from different viewpoints within a predetermined time error range relative to a time specified by the user 91 from the satellite management device 11. These multiple satellite images from different viewpoints corresponding to times within the predetermined time error range relative to the specified time are referred to as quasi-real-time satellite images to distinguish them from the satellite images used to generate the base image and the real-time satellite images. Although the error range of several hours relative to the specified time is an assumed error range in constellation operation, the error range relative to the specified time of the quasi-real-time satellite images is set to a maximum of one day, assuming the observed satellites have a return date.

[0130] The real-time image generation unit 72 extracts dynamic subjects on the ground from each of the multiple acquired near-real-time satellite images and generates 3D models of the dynamic subjects. The extraction of dynamic subjects and the generation of 3D models of dynamic subjects are the same as in the case of real-time satellite images.

[0131] Next, the real-time image generation unit 72 estimates the 3D model of the moving subject at a specified time based on the generated 3D model of the moving subject and using external information about the moving subject obtained from the external information providing server 102 via the external information acquisition unit 73. For example, the real-time image generation unit 72 acquires the position information of an aircraft or ship as external information and moves the 3D model of the moving object to the position based on the external information. Furthermore, for example, the real-time image generation unit 72 acquires cloud distribution information as external information and moves the 3D model of the clouds to the position based on the external information. Then, the real-time image generation unit 72 generates a second virtual viewpoint image of the moving subject as a real-time image of the moving subject when viewing the estimated 3D model of the moving subject at the specified time based on external information from a predetermined virtual viewpoint specified by the user 91, and provides this real-time image to the image synthesis unit 74.

[0132] Alternatively, the real-time image generation unit 72 generates a near-real-time image of the dynamic subject when viewed from a predetermined virtual viewpoint specified by the user 91, based on a 3D model of the dynamic subject generated from multiple acquired near-real-time satellite images. Then, the real-time image generation unit 72 uses the generated near-real-time image of the dynamic subject and external information about the dynamic subject acquired from the external information providing server 102 via the external information acquisition unit 73 to estimate and generate a real-time image of the dynamic subject. For example, a near-real-time image of the texture of an aircraft or ship two hours ago is changed to a location at a specified time based on external information and used as a real-time image of a moving object at that specified time. A near-real-time image of the texture of clouds two hours ago is changed based on information about the clouds at a specified time based on external information, thus generating a real-time image of the clouds at that specified time.

[0133] In other words, when acquiring near-real-time satellite images, the real-time image of the dynamic subject is generated through the collaboration (complementary relationship) of two methods: acquiring real-time satellite images to generate real-time images of the dynamic subject and generating real-time images of the dynamic subject using only external information.

[0134] Even when real-time satellite imagery corresponding to the time specified by user 91 can be acquired, quasi-real-time satellite imagery can also be used. By using both real-time and quasi-real-time satellite imagery together, the following effects can be achieved. As an effect, even images with less information in the real-time satellite imagery captured at the specified time are permissible. For example, by using both real-time and quasi-real-time satellite imagery, the resolution of the real-time satellite imagery captured at the specified time may be lower, and the imaging range can be extended for lower resolution. Different types of satellites 21 can be used as satellites 21 for capturing real-time satellite imagery and satellites 21 for capturing quasi-real-time satellite imagery, thus providing space for preparing multiple types of satellites 21. Based on the difference information between real-time and quasi-real-time satellite imagery, information between them, such as changes in clouds over a time period, can also be supplemented.

[0135] Furthermore, the real-time image generation unit 72 can use a base image that can be generated by the base image generation unit 71 to estimate the lighting information of a moving subject and generate a real-time satellite image of the lighting information. For example, the base image generation unit 71 can generate a 3D model of a stationary subject under each condition (e.g., each at different times (e.g., morning, noon, and evening), each in clear weather and sunny weather, or each season (e.g., spring, summer, autumn, or winter)) to generate a base image for each condition. The real-time image generation unit 72 can detect changes in the lighting information in the base image for each condition generated based on the 3D model of the stationary subject under each condition and generate a real-time satellite image based on the time, season, and weather specified by the user 91 based on the estimation. The real-time satellite image can be estimated using quasi-real-time satellite images or incomplete real-time satellite images. Incomplete real-time satellite images are, for example, panchromatic images (monochrome images) or low-resolution images. As with lighting information, texture information representing the shadows and colors of moving objects can also be estimated using the base image.

[0136] Image synthesis unit 74 synthesizes a base image provided by base image generation unit 71 and a real-time satellite image provided by real-time image generation unit 72 to generate a free-viewpoint image of a specified location on the ground at a time specified by user 91, and provides the free-viewpoint image to free-viewpoint image accumulation unit 62. In image synthesis unit 74, the real-time image corresponding to the foreground image (subject) of volumetric capture technology is superimposed on the base image corresponding to the background image of volumetric capture technology.

[0137] The free viewpoint image accumulation unit 62 accumulates free viewpoint images provided by the image synthesis unit 74. The free viewpoint image accumulation unit 62 can accumulate various virtual viewpoints, free viewpoint images at specified times and locations generated by the free viewpoint image generation unit 61, select a specified free viewpoint image in response to an instruction from the control unit 66, and provide the selected free viewpoint image to the encoding unit 63 or the display unit 81.

[0138] Encoding unit 63 encodes the free-viewpoint image provided by free-viewpoint image accumulation unit 62 using a predetermined encoding method such as Advanced Video Coding (AVC) or High Efficiency Video Coding (HEVC). The encoded free-viewpoint image data is then provided to communication unit 64.

[0139] Communication unit 64 communicates with terminal device 29 via a predetermined network. When terminal device 92 provides a generation instruction for generating free-viewpoint images at a predetermined virtual viewpoint, specified time, and specified location, communication unit 64 provides the generation instruction to control unit 66. In response to the generation instruction, communication unit 64 sends the encoded free-viewpoint image data provided by encoding unit 63 to terminal device 92. Furthermore, under the control of control unit 66, communication unit 64 sends a satellite image acquisition request to satellite management device 11. Communication unit 64 also communicates with satellite image accumulation server 101 and external information providing server 102 under the control of control unit 66.

[0140] Display unit 81 is configured as, for example, a liquid crystal display (LCD) or an organic electroluminescent (EL) display. Display unit 81 displays a free-viewpoint image provided by free-viewpoint image accumulation unit 62.

[0141] The operation unit 82 is configured, for example, to be a keyboard, mouse, touch panel, etc., to receive operations from the user 91, and to provide the control unit 66 with a generation instruction for generating a free viewpoint image at a specified time and location specified by the user 91.

[0142] The control unit 66 controls the operation of the entire free-viewpoint image generation device 51. For example, the control unit 66 provides the free-viewpoint image generation unit 61 with an instruction to generate a free-viewpoint image of a predetermined location on the ground viewed from a predetermined virtual viewpoint in the sky specified by the user 91 at a predetermined time, based on a generation instruction from the communication unit 64 or the user IF unit 65. Furthermore, when it is necessary to capture satellite images required for generating the free-viewpoint image, the control unit 66 causes the communication unit 64 to send a satellite image capture request to the satellite management device 11. Additionally, the control unit 66 provides the free-viewpoint images accumulated in the free-viewpoint image accumulation unit 62 to the encoding unit 63 or the display unit 81.

[0143] Terminal device 92 is configured as, for example, a smartphone, tablet computer terminal, mobile phone, personal computer, etc., and receives operations from user 91. When user 91 instructs free-viewpoint image generation device 51 to generate a free-viewpoint image via terminal device 92, terminal device 92 sends a generation instruction to free-viewpoint image generation device 51. Furthermore, terminal device 92 receives free-viewpoint image encoded data sent from free-viewpoint image generation device 51 in response to the generation instruction, performs decoding processing corresponding to the encoding method, and displays the free-viewpoint image on a display device (not shown). The user can view the free-viewpoint image remotely from anywhere using the terminal device 92 owned by the user.

[0144] <4. Free Viewpoint Image Generation and Processing>

[0145] Next, we will refer to Figure 3 The flowchart describes the free-viewpoint image generation process performed by the free-viewpoint image generation device 51. For example, the process begins when user 91 performs an operation via terminal device 92 to instruct the generation of a free-viewpoint image of a predetermined location on the ground viewed from a predetermined virtual viewpoint in the sky at a predetermined time.

[0146] First, in step S1, the control unit 66, based on the generation instruction from the terminal device 92, sends a request via the communication unit 64 to the satellite management device 11 for capturing satellite images corresponding to a predetermined virtual viewpoint, a specified time, and a specified location. Furthermore, the control unit 66 provides an instruction to the free-viewpoint image generation unit 61 for generating a free-viewpoint image of a predetermined location on the ground viewed from a predetermined virtual viewpoint in the sky specified by the user 91 at a predetermined time.

[0147] In step S2, the basic image generation unit 71 acquires multiple satellite images captured from different viewpoints and accumulated in the satellite image accumulation server 101, and removes moving subjects included in each of the multiple satellite images.

[0148] In step S3, the base image generation unit 71 uses multiple satellite images of a stationary subject on the ground, after the dynamic subject has been removed, to generate a 3D model. Then, in step S4, the base image generation unit 71 generates a first virtual viewpoint image as a base image when the generated 3D model of the stationary subject is viewed from a predetermined virtual viewpoint specified by the user 91, and provides the first virtual viewpoint image to the image synthesis unit 74.

[0149] In step S5, the real-time image generation unit 72, in response to the capture request in step S1, determines whether multiple real-time satellite images can be acquired from the satellite management device 11. These real-time satellite images correspond to a specified location and time specified by the user 91 and have been captured from different viewpoints.

[0150] If it is determined in step S5 that multiple real-time satellite images captured from different viewpoints can be acquired, the process proceeds to step S6, and the real-time image generation unit 72 extracts the dynamic subject from each of the multiple acquired real-time satellite images.

[0151] In step S7, the real-time image generation unit 72 generates a 3D model of the dynamic subject based on each real-time satellite image that includes only the extracted dynamic subject on the ground. Then, in step S8, the real-time image generation unit 72 generates a second virtual viewpoint image as a real-time image of the dynamic subject when viewing the 3D model of the dynamic subject from a predetermined virtual viewpoint specified by the user 91, and provides the second virtual viewpoint image to the image synthesis unit 74.

[0152] On the other hand, if it is determined in step S5 that multiple real-time satellite images captured from different viewpoints cannot be obtained, the process proceeds to step S9, in which the real-time image generation unit 72 determines whether multiple quasi-real-time satellite images can be obtained from the satellite management device 11. These quasi-real-time satellite images are satellite images captured at a time within the error range of a predetermined time relative to a specified time specified by the user 91 and captured from different viewpoints.

[0153] If it is determined in step S9 that multiple near-real-time satellite images captured from different viewpoints can be obtained from the satellite management device 11, the process proceeds to step S10, and the real-time image generation unit 72 extracts a dynamic subject from each of the multiple acquired near-real-time satellite images, and generates a 3D model of the dynamic subject based on each near-real-time satellite image that includes only the extracted dynamic subject on the ground.

[0154] In step S11, the real-time image generation unit 72 obtains external information about the dynamic subject from the external information providing server 102 via the external information acquisition unit 73.

[0155] In step S12, the real-time image generation unit 72 uses the acquired external information about the moving subject and estimates the 3D model of the moving subject at a specified time based on the generated 3D model of the moving subject.

[0156] In step S13, the real-time image generation unit 72 generates a second virtual viewpoint image as a real-time image of the dynamic subject when viewing the estimated 3D model of the dynamic subject at a specified time from a predetermined virtual viewpoint specified by the user 91, and provides the second virtual viewpoint image to the image synthesis unit 74.

[0157] On the other hand, if it is determined in step S9 that multiple near real-time satellite images captured from different viewpoints cannot be obtained from the satellite management device 11, the process proceeds to step S14, and the real-time image generation unit 72 obtains external information about the dynamic subject from the external information providing server 102 via the external information acquisition unit 73.

[0158] In step S15, the real-time image generation unit 72 uses the acquired external information to generate a real-time image of the dynamic subject and provides it to the image synthesis unit 74.

[0159] In step S16, the image synthesis unit 74 synthesizes the base image provided by the base image generation unit 71 and the real-time satellite image provided by the real-time image generation unit 72 to generate a free viewpoint image of a specified location on the ground at a time specified by the user 91, and provides the free viewpoint image to the free viewpoint image accumulation unit 62.

[0160] In step S17, the free viewpoint image accumulation unit 62 accumulates the free viewpoint image provided by the image synthesis unit 74, and provides the free viewpoint image provided by the image synthesis unit 74 to the encoding unit 63 under the control of the control unit 66.

[0161] In step S18, the encoding unit 63 encodes the free viewpoint image provided by the free viewpoint image accumulation unit 62 using a predetermined encoding method, and the communication unit 64 sends the encoded free viewpoint image data to the terminal device 92. The free viewpoint image based on the encoded free viewpoint image data is displayed on the terminal device 92.

[0162] When a generation instruction is provided from the operation unit 82, the processing of steps S17 and S18 is changed to the following: the free viewpoint image provided from the image synthesis unit 74 is provided to the display unit 81, and the free viewpoint image is displayed on the display unit 18.

[0163] In this way, the free viewpoint image generation process is complete.

[0164] In the above free-viewpoint image generation process, steps S2 and S3 for generating the 3D model of the stationary subject do not affect the real-time situation at the predetermined time and location specified by user 91, and therefore can be performed in conjunction with... Figure 3 The free viewpoint image generation process is pre-executed at different times.

[0165] In the above-described free-viewpoint image generation process, when multiple real-time satellite images corresponding to a specified location and time specified by user 91 can be acquired from satellite management device 11, free-viewpoint image generation unit 61 uses the acquired multiple real-time satellite images to generate a 3D model of the dynamic subject. The condition for acquiring multiple real-time satellite images corresponding to a specified location and time is limited to a certain time period during which satellite 21 passes over the specified location, and if satellite 21 is a low-orbit satellite, the most desirable condition is approximately 10 minutes around the specified time relative to the specified time; however, if a range that can be estimated from the captured satellite images based on time errors is included, it is approximately 1 hour before and after the specified time (inclusive of 30 minutes).

[0166] On the other hand, if multiple real-time satellite images cannot be acquired within a time range of several hours (more than approximately one hour) relative to a specified location and time, a real-time image of the moving subject is generated based on multiple near-real-time satellite images. In this case, there exists a first estimation method that estimates a temporary 3D model of the moving subject and generates a real-time image of the moving subject, and a second estimation method that generates a near-real-time image of the temporary moving subject and estimates a real-time image of the moving subject. More specifically, in the first estimation method, the free-viewpoint image generation unit 61 generates a 3D model of the moving subject based on multiple near-real-time satellite images within an error range of several hours from the specified time, and uses external information to estimate the 3D model of the moving subject at the specified time to generate a real-time image of the moving subject. In the second estimation method, the free viewpoint image generation unit 61 generates a temporary dynamic subject 3D model based on multiple near-real-time satellite images within an error range of several hours from a specified time, uses the temporary dynamic subject 3D model to generate a near-real-time image of the dynamic subject from a virtual viewpoint specified by the user 91, and uses external information to estimate and generate a real-time image of the dynamic subject based on the near-real-time image of the dynamic subject.

[0167] Furthermore, during periods when even near-real-time satellite imagery is unavailable, real-time images of the dynamic subject are generated using real-time external information obtained from the external information providing server 102 and known information about the dynamic subject. For example, a real-time image of the dynamic subject is generated based on real-time acquired aircraft position information (AIS information) and texture information generated from a known 3D model of the aircraft. While multiple real-time or near-real-time satellite images are required to generate a 3D model of the dynamic subject, only one satellite image may be needed when using 2D satellite imagery as real-time external information.

[0168] By appropriately combining available real-time satellite imagery, near-real-time satellite imagery, or external information, it is possible to identify moving subjects or generate real-time images without requiring... Figure 3 In the free-viewpoint image generation process, the situation is divided into three cases: the case where real-time satellite imagery can be acquired (steps S6 to S8), the case where near-real-time satellite imagery can be acquired (steps S10 to S13), and the case where neither real-time nor near-real-time satellite imagery can be acquired (steps S14 and S15). As described above, for example, effects such as obtaining real-time satellite imagery with low resolution and a wide imaging range can also be achieved by combining real-time and near-real-time satellite imagery.

[0169] Real-time satellite images corresponding to a specified location and time, near-real-time satellite images, real-time external information obtained from external information providing server 102, and known information about the dynamic subject used to generate real-time images of the dynamic subject can be dynamic subject identification information for identifying the dynamic subject.

[0170] The free-viewpoint image generation unit 61 uses satellite images captured by satellite 21 and a 3D model of a stationary subject generated by recognizing the dynamic subject to identify the dynamic subject to generate a free-viewpoint image of a predetermined location on the ground viewed from a predetermined virtual viewpoint in the sky. Therefore, any location on the ground can be observed from a free-viewpoint (virtual viewpoint) in the sky at any time.

[0171] It is anticipated that the limited number of satellites 21 capable of performing imaging at a specified location and time as specified by user 91 will be resolved by increasing the number of operational satellites and developing communication technologies for inter-satellite communication.

[0172] Furthermore, when user 91 makes a reservation in advance and specifies a predetermined location and time, and desires to view satellite images from a predetermined virtual viewpoint, the free-viewpoint image generation device 51 can request the satellite management device 11 to operate satellite 21 so that satellite 21 passes over the predetermined location at the specified time, and control the satellite management device 11 to pre-deploy the satellite 21 to be imaged and to instruct it on orbital transitions. Therefore, the free-viewpoint image generation device 51 can acquire satellite images at the predetermined location and time, and user 91 can observe the free-viewpoint images. The free-viewpoint image generation device 51 can obtain orbital information about satellite 21 from the satellite management device 11, etc., and present the user 91 with locations and times where free-viewpoint images can be generated, from which user 91 can select a location and time and observe the free-viewpoint images.

[0173] Part of the processing performed by the free-viewpoint image generation device 51 can be performed by the satellite 21.

[0174] For example, satellite 21 has a real-time image generation function performed by real-time image generation unit 72, generating real-time images viewed from a predetermined virtual viewpoint at a specified location and time by user 91, and performing downlink transmission of the real-time images. When obtaining real-time images by extracting dynamic subjects on the ground based on differences from a base image at the same viewpoint, a base image needs to be accumulated in satellite 21. Free-viewpoint image generation device 51 acquires the real-time images generated by satellite 21 via satellite management device 11 and combines them with its own generated base images to generate free-viewpoint images. In this case, the amount of data required to generate real-time images to be transmitted from satellite 21 to ground station 15 can be reduced. By downlink transmission of a 3D model of a dynamic subject from satellite 21 to ground station 15, the generation of real-time images can be performed by the free-viewpoint image generation device 51 on the ground. In this case, the amount of downlink data required to generate real-time images can also be reduced.

[0175] <5. Application Examples>

[0176] The satellite image processing system 1 described above can be applied to the following applications.

[0177] • Inspection of landmarks from a free viewpoint

[0178] By generating free-viewpoint images of a user's travel destination or when flying over landmarks such as Mount Fuji, users can enjoy a perspective from space.

[0179] Inspections of vast areas such as disaster zones

[0180] By generating free-viewpoint images of disaster or conflict zones, the situation in those zones can be confirmed.

[0181] • Satellite images with 3D effects

[0182] By presenting satellite images obtained through satellite remote sensing, which are used to observe the condition of a target area or object and detect changes in the condition, as free-viewpoint images, natural images with a three-dimensional effect can be viewed even from a macroscopic viewpoint.

[0183] Advanced meteorological observation

[0184] By using free-viewpoint images to perform three-dimensional examination of meteorological phenomena from a free viewpoint, more advanced analysis of meteorological phenomena such as clouds can be achieved. Furthermore, the analysis results can be used for advanced weather forecasting and other applications.

[0185] • Determine the growth status of crops and trees

[0186] Free-viewpoint images can be used to present the growth status of crops and trees as 3D visual data. Because it is a free-viewpoint image, more diverse analyses can be performed.

[0187] ·Urban Plan

[0188] Free-viewpoint imagery can be used to observe the city's condition in three dimensions. In particular, real-time changes can be observed. In this case, points of change in the city can be visualized by highlighting the changed portions relative to the base image. Real-time information (vehicle movement, traffic lights, etc.) near a specific time can be determined from multiple viewpoints.

[0189] • Determine the condition of the construction site

[0190] Free-viewpoint imagery can be used to determine the condition of facilities in remote areas in three dimensions. Facilities include buildings such as houses and towers, dams, petrochemical plants, factories, ports, etc. Furthermore, free-viewpoint imagery can be used to determine land maintenance conditions in three dimensions, such as leveling. In this case, points of change in a city can be visualized by highlighting changed portions relative to the base image. Real-time information (vehicle movement, guide lines, etc.) near a specific time can be determined from multiple viewpoints.

[0191] Furthermore, applications that can associate free-viewpoint images generated by the satellite image processing system 1 with images captured by other imaging devices are conceivable. For example, a system capable of switching an image from a free-viewpoint image generated by the free-viewpoint image generation device 51 to an image captured from the sky at close range is conceivable. When a user examines a free-viewpoint image as a macro image of a target location and performs a zoom operation on a portion of the free-viewpoint image, the user can simultaneously examine a detailed image of a portion of the free-viewpoint image by associating the free-viewpoint image with an image captured from the sky at close range, thereby switching the free-viewpoint image to an image captured by a surveillance camera or drone in the city.

[0192] <6. Computer Configuration Example>

[0193] The series of processes performed by the free-viewpoint image generation device 51 can be executed by hardware or software. When this series of processes is executed by software, the program constituting the software is installed in a computer. Here, the computer includes a microcomputer embedded in dedicated hardware, or, for example, a general-purpose personal computer capable of performing various functions by installing various programs.

[0194] Figure 4 This is a block diagram illustrating an example of a computer hardware configuration performing a series of processes executed by the free-viewpoint image generation device 51.

[0195] In a computer, the central processing unit (CPU) 301, read-only memory (ROM) 302, and random access memory (RAM) 303 are connected to each other via bus 304.

[0196] The input / output interface 305 is further connected to the bus 304. The input unit 306, output unit 307, storage unit 308, communication unit 309, and driver 310 are connected to the input / output interface 305.

[0197] Input unit 306 is, for example, a keyboard, mouse, microphone, touch panel, or input terminal. Output unit 307 is, for example, a display, speaker, or output terminal. Storage unit 308 is, for example, a hard disk, RAM disk, or non-volatile memory. Communication unit 309 is a network interface, etc. Driver 310 drives removable recording medium 311, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0198] In a computer with the above configuration, the CPU 301 performs the aforementioned series of processes, for example, by loading a program stored in the storage unit 308 into the RAM 303 via the input / output interface 305 and the bus 304 and executing the program. The RAM 303 also appropriately stores data required by the CPU 301 to perform various processes.

[0199] The program executed by the computer (CPU 301) can be recorded on a removable recording medium 311, such as a packaging medium used for supply. The program can be provided via wired or wireless transmission media, such as a local area network, the Internet, or digital satellite broadcasting.

[0200] In a computer, a program can be installed in a storage unit 308 via an input / output interface 305 by mounting a removable recording medium 311 on a drive 310. The program can be received by a communication unit 309 via a wired or wireless transmission medium for installation in the storage unit 308. Alternatively, the program can be pre-installed in a ROM 302 or in the storage unit 308.

[0201] Note that in this specification, even if these steps are not performed in the order described in the flowchart in time sequence, and when these steps are performed in time sequence, the steps described in the flowchart may be performed in parallel or at the required time intervals (e.g., when invoked).

[0202] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), and all components may or may not be housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a device in which multiple modules are housed in one housing, can both be considered a system.

[0203] The embodiments of this technology are not limited to the above embodiments, and various changes can be made without departing from the essential points of this technology.

[0204] For example, a combination of all or part of the above embodiments may be used.

[0205] For example, this technology can have a cloud computing configuration, in which one function is shared and processed jointly by multiple devices via a network.

[0206] Furthermore, each step described in the flowchart above can be performed by one device or shared by multiple devices.

[0207] Furthermore, in cases where a step includes multiple processes, the multiple processes included in a step can be executed by a single device, or can be shared and executed by multiple devices.

[0208] The effects described in this specification are merely illustrative and not limited, and effects other than those described in this specification may exist.

[0209] This technology can be configured as follows. (1)

[0211] An image generation device includes an image generation unit that uses a 3D model of a stationary subject generated from satellite images captured by an artificial satellite and dynamic subject identification information to generate a free-viewpoint image of a predetermined location on the ground viewed from a predetermined virtual viewpoint in the sky. (2)

[0213] According to the image generation device of (1), wherein the image generation unit generates a 3D model of a static subject by using a base image viewed from a predetermined virtual viewpoint in the sky as a background image, generates a real-time image viewed from a predetermined virtual viewpoint in the sky as a foreground image by using dynamic subject recognition information, and synthesizes the base image and the real-time image to generate a free viewpoint image. (3)

[0215] According to the image generation device described in (2), the image generation unit uses a 3D model of a dynamic subject as dynamic subject recognition information. (4)

[0217] According to the image generation device described in (3), the image generation unit generates a real-time image of a 3D model of a dynamic subject viewed from a predetermined virtual viewpoint in the sky as a foreground image. (5)

[0219] The image generation device according to (3) or (4) wherein the image generation unit uses real-time satellite images captured by artificial satellites near a time specified by the user to generate a 3D model of a dynamic subject. (6)

[0221] According to the image generation device described in (5), the image generation unit extracts a dynamic subject from a real-time satellite image and generates a 3D model of the dynamic subject based on the extracted image. (7)

[0223] According to the image generation device described in (3), the image generation unit uses a near-real-time satellite image captured by an artificial satellite at a time several hours different from the time specified by the user to generate a 3D model of a temporary dynamic subject, and uses external information to estimate the 3D model of the dynamic subject at the time specified by the user based on the 3D model of the temporary dynamic subject to generate a real-time image. (8)

[0225] According to the image generation device described in (3), the image generation unit uses a near-real-time satellite image captured by an artificial satellite at a time difference of several hours from a time specified by the user to generate a 3D model of a temporary dynamic subject, generates a near-real-time image of the dynamic subject when viewed from a predetermined virtual viewpoint in the sky, and uses external information to generate a real-time image based on the near-real-time image of the dynamic subject. (9)

[0227] The image generation device according to (7) or (8) wherein the image generation unit extracts a dynamic subject from a near real-time satellite image and generates a temporary 3D model of the dynamic subject based on the extracted image. (10)

[0229] According to the image generation device described in (2), the image generation unit uses real-time external information and known information about the dynamic subject as dynamic subject identification information. (11)

[0231] The image generation apparatus according to any one of (1) to (10), wherein the image generation unit uses an image of a satellite image captured by a satellite in which a dynamic subject has been removed to generate a 3D model of a stationary subject. (12)

[0233] An image generation method includes generating a free-viewpoint image of a predetermined location on the ground from a predetermined virtual viewpoint in the sky using a 3D model of a stationary subject generated from satellite images captured by an artificial satellite and dynamic subject identification information that identifies a dynamic subject. (13)

[0235] A program that uses a computer as an image generation unit, the image generation unit using a 3D model of a stationary subject generated from satellite images captured by an artificial satellite and dynamic subject identification information that identifies a moving subject, to generate a free-viewpoint image of a predetermined location on the ground viewed from a predetermined virtual viewpoint in the sky.

[0236] Reference tag list

[0237] 1. Satellite Image Processing System

[0238] 11 Satellite Management Equipment

[0239] 21 satellites

[0240] 51 Free-viewpoint image generation device

[0241] 61 Free Viewpoint Image Generation Unit

[0242] 62 Free-viewpoint image accumulation units

[0243] 63 coding units

[0244] 65 User IF Unit

[0245] 66 Control Unit

[0246] 71 Basic Image Generation Unit

[0247] 72 Real-time Image Generation Units

[0248] 73 External Information Acquisition Unit

[0249] 74 Image Composition Units

[0250] 101 Satellite Image Accumulation Server

[0251] 102 External Information Provider Server

[0252] 301 CPU

[0253] 302 ROM

[0254] 303 RAM

[0255] 306 Input Unit

[0256] 307 Output Unit

[0257] 308 memory cells

[0258] 309 Communication Unit

[0259] 310 drive

Claims

1. An image generation apparatus including an image generation unit that generates a free viewpoint image that views a predetermined place on the ground from a predetermined virtual viewpoint in the sky using a 3D model of a stationary subject generated using a satellite image captured by an artificial satellite and dynamic subject recognition information that recognizes a dynamic subject, wherein the image generation unit generates the free viewpoint image by generating the 3D model of the stationary subject using a base image viewed from the predetermined virtual viewpoint in the sky as a background image, generating a real-time image that views the predetermined virtual viewpoint in the sky using the dynamic subject recognition information as a foreground image, and compositing the base image and the real-time image; and wherein the image generation unit uses real-time external information and known information about the dynamic subject as the dynamic subject recognition information.

2. The image generation device of claim 1, wherein, the image generation unit also uses a 3D model of the dynamic subject as the dynamic subject recognition information.

3. The image generation device of claim 2, wherein, the image generation unit generates a real-time image of the 3D model of the dynamic subject viewed from the predetermined virtual viewpoint in the sky as the foreground image.

4. The image generation device of claim 2, wherein, the image generation unit generates the 3D model of the dynamic subject using a real-time satellite image captured by the artificial satellite near a time specified by a user.

5. The image generation device of claim 4, wherein, the image generation unit extracts the dynamic subject from the real-time satellite image and generates the 3D model of the dynamic subject from the extracted image.

6. The image generation device of claim 2, wherein, the image generation unit generates a 3D model of a temporary dynamic subject using a quasi-real-time satellite image captured by the artificial satellite at a time that differs from the time specified by the user by several hours, and estimates a 3D model of the dynamic subject at the time specified by the user from the 3D model of the temporary dynamic subject using external information to generate the real-time image.

7. The image generation device of claim 2, wherein, the image generation unit generates a 3D model of a temporary dynamic subject using a quasi-real-time satellite image captured by the artificial satellite at a time that differs from the time specified by the user by several hours, generates a quasi-real-time image of the dynamic subject obtained when the 3D model of the temporary dynamic subject is viewed from the predetermined virtual viewpoint in the sky, and generates the real-time image from the quasi-real-time image of the dynamic subject using external information.

8. The image generation device of claim 6, wherein, the image generation unit extracts the dynamic subject from the quasi-real-time satellite image and generates the 3D model of the temporary dynamic subject from the extracted image.

9. The image generation device of claim 1, wherein, the image generation unit generates the 3D model of the stationary subject using an image in which the dynamic subject included in the satellite image captured by the artificial satellite has been removed.

10. An image generation method including generating, by an image generation apparatus, a free viewpoint image that views a predetermined place on the ground from a predetermined virtual viewpoint in the sky using a 3D model of a stationary subject generated using a satellite image captured by an artificial satellite and dynamic subject recognition information that recognizes a dynamic subject, wherein the image generation apparatus generates the free viewpoint image by generating the 3D model of the stationary subject using a base image viewed from the predetermined virtual viewpoint in the sky as a background image, generating a real-time image that views the predetermined virtual viewpoint in the sky using the dynamic subject recognition information as a foreground image, and compositing the base image and the real-time image; and wherein the image generation apparatus uses real-time external information and known information about the dynamic subject as the dynamic subject recognition information. wherein the image generation device uses real-time external information and known information about the dynamic subject as dynamic subject identification information.

11. A program product for causing a computer to function as an image generation unit that generates a free view point image of a predetermined place on the ground viewed from a predetermined virtual view point in the sky using a 3D model of a static subject generated using a satellite image captured by an artificial satellite and dynamic subject identification information that identifies a dynamic subject, wherein the image generation unit generates the free view point image by generating the 3D model of the static subject using a base image viewed from the predetermined virtual view point in the sky as a background image, generating a real-time image viewed from the predetermined virtual view point in the sky using the dynamic subject identification information as a foreground image, and compositing the base image and the real-time image; and wherein the image generation unit uses real-time external information and known information about the dynamic subject as dynamic subject identification information. ​

Images