Image management method

Abstract

The present technology pertains to: an image management method enabling management of an image captured by an artificial satellite; and a metadata data structure. In the present technology, a management device for managing an image captured by a satellite adds, to the captured image, metadata at least including relevant-person information about the captured image. The present technology is applicable to, for example, an artificial satellite or the like that performs satellite remote sensing through formation flight.

Description

[Technical field]

[0001] This technology is image management method In particular, an image management system that can manage images captured by artificial satellites method Regarding. [Background technology]

[0002] There is a technology called satellite remote sensing that observes the condition of a target area or object and detects changes in the condition from images of a specific area on the earth captured by an artificial satellite (see, for example, Patent Documents 1 and 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2010 / 097921 [Patent Document 2] JP 2004-15451 A Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, research and development of low-cost small satellites has been progressing by private companies, and it is expected that many images will be taken by satellites in the future, so there is a demand for a system to manage the images taken by satellites.

[0005] The present technology has been made in consideration of such circumstances, and makes it possible to manage images captured by artificial satellites. [Means for solving the problem]

[0006] The image management method according to the first aspect of the present technology includes: Operated by formation flights A management device for managing images captured by a satellite, A constellation identifier for identifying a constellation of satellites in a formation flight including the satellite; and a satellite identifier for identifying the satellite. Information about the person involved in the captured image and Metadata including at least the above is added to the captured image.

[0007] In a first aspect of the present technology, Operated by formation flights satellite A constellation identifier for identifying a constellation of satellites of a formation flight including: Information on the people involved in the captured image and The metadata including at least the above is added to the captured image. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example configuration of a first embodiment of a satellite image processing system to which the present technology is applied. [Diagram 2] FIG. 1 is a diagram illustrating a formation flight. [Diagram 3] FIG. 1 is a diagram illustrating a formation flight. [Figure 4] FIG. 2 is a block diagram showing an example of the configuration of a satellite. [Diagram 5] 2 is a block diagram showing an example configuration of a satellite group management device, a communication device, and an image analysis server. FIG. [Figure 6] 11 is a flowchart illustrating an imaging sequence focusing on one satellite. [Figure 7] 7 is a detailed flowchart of the imaging preparation process in step S33 of FIG. 6. [Figure 8] FIG. 4 is a diagram illustrating a determination of a remaining battery charge. [Figure 9] 1 is a flowchart of a satellite image processing system performing a formation flight. [Figure 10] FIG. 2 is a diagram for explaining information added as metadata. [Figure 11] FIG. 11 is a diagram illustrating a configuration example of a second embodiment of a satellite image processing system to which the present technology is applied. [Figure 12] FIG. 13 is a block diagram showing an example of the configuration of a transmission device in a second embodiment. [Figure 13] 13 is a flowchart illustrating a first event imaging sequence by the satellite image processing system of the second embodiment. [Figure 14]10 is a flowchart illustrating a second event imaging sequence by the satellite image processing system of the second embodiment. [Figure 15] 13 is a flowchart illustrating a third event imaging sequence by the satellite image processing system of the second embodiment. [Figure 16] FIG. 13 is a block diagram showing another example configuration of the transmission device in the second embodiment. [Figure 17] 1 is a block diagram showing an example of the configuration of an embodiment of a computer to which the present technology is applied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, modes for carrying out the present technology (hereinafter, referred to as embodiments) will be described in the following order. 1. Example of satellite image processing system configuration 2. Single-unit imaging sequence 3. Preparation for Imaging 4. Formation Flight Flowchart 5. Image processing examples 6. Metadata details 7.Details of distribution management process 8. Examples of Formation Flight Applications 9. Second embodiment of satellite image processing system 10. First event imaging sequence of the second embodiment 11. Second event imaging sequence of the second embodiment 12. Third event imaging sequence of the second embodiment 13. Other configuration examples of the transmitting device 14. Application example of satellite image processing system using event detection sensor 15. Computer configuration example

[0012] <1. Example of satellite image processing system configuration> FIG. 1 is a block diagram showing an example of the configuration of a first embodiment of a satellite image processing system to which the present technology is applied.

[0013] The satellite image processing system 1 in Fig. 1 is a system that performs satellite remote sensing to observe the conditions of target areas or objects on the earth and detect changes in conditions using images captured by multiple artificial satellites (hereinafter simply referred to as satellites). In this embodiment, the satellite is equipped with an imaging device and has at least the function of capturing images of the earth.

[0014] The satellite operating company has a satellite group control device 11 that manages multiple satellites 21, and multiple communication devices 13 that communicate with the satellites 21. Note that the satellite group control device 11 and some of the multiple communication devices 13 may be devices owned by parties other than the satellite operating company. The satellite group control device 11 and the multiple communication devices 13 are connected via a predetermined network 12. The communication devices 13 are disposed in a ground station (terrestrial base station) 15. Note that, although an example in which the number of communication devices 13 is three, communication devices 13A to 13C, is shown in FIG. 1, the number of communication devices 13 is arbitrary.

[0015] The satellite group management device 11 manages a plurality of satellites 21 owned by a satellite operating company. Specifically, the satellite group management device 11 acquires related information from one or more information providing servers 41 of external organizations as necessary, and determines an operation plan for the plurality of satellites 21 owned by the satellite group management device 11. Then, the satellite group management device 11 instructs a specific satellite 21 to capture an image via the communication device 13 in response to a request from a customer, thereby causing the specific satellite 21 to capture an image. The satellite group management device 11 also acquires and stores captured images transmitted from the satellite 21 via the communication device 13. The acquired captured images are subjected to a predetermined image processing as necessary, and are provided (transmitted) to the customer. Alternatively, the acquired captured images are provided (transmitted) to an image analysis server 42 of an image analysis company, and are provided to the customer after undergoing a predetermined image processing.

[0016] The information providing server 41 installed at an external organization supplies predetermined related information to the satellite constellation management device 11 via a predetermined network in response to a request from the satellite constellation management device 11 or periodically. Examples of related information provided by the information providing server 41 include the following. For example, satellite orbit information described in TLE (Two Line Elements) format can be obtained as related information from NORAD (North American Aerospace Defense Command) as an external organization. Also, for example, meteorological information such as the weather and cloud cover at a predetermined point on the earth can be obtained from a weather information provider as an external organization.

[0017] The image analysis server 42 performs predetermined image processing on the images captured by the satellites 21, which are supplied from the satellite group management device 11 via a predetermined network. The processed images are provided to the clients of the image analysis company, or supplied to the satellite group management device 11 of the satellite operating company. For example, the image analysis server 42 performs metadata generation processing for adding predetermined metadata to the images captured by the satellites 21, correction processing such as distortion correction of the captured images, image synthesis processing such as color synthesis processing, etc. Image processing of the captured images may be performed by the satellite operating company, in which case the satellite operating company and the image analysis company are the same. The satellite group management device 11 and the image analysis server 42 may also be realized in a single device.

[0018] The communication device 13 communicates with a specific satellite 21 designated by the satellite group management device 11 via the antenna 14 under the control of the satellite group management device 11. For example, the communication device 13 transmits an imaging instruction to the specific satellite 21 to image a specific area on the ground at a specific time and position. The communication device 13 also receives captured images transmitted from the satellite 21 and supplies the captured images to the satellite group management device 11 via the network 12. Transmission from the communication device 13 of the ground station 15 to the satellite 21 is also called an uplink, and transmission from the satellite 21 to the communication device 13 is also called a downlink. The communication device 13 can communicate directly with the satellite 21, and can also communicate via a relay satellite 22. For example, a geostationary satellite is used as the relay satellite 22.

[0019] The network 12 and the network between the information providing server 41 or the image analysis server 42 and the satellite group management device 11 may be any communication network, and may be a wired communication network, a wireless communication network, or both. In addition, the network 12 and the network between the information providing server 41 or the image analysis server 42 and the satellite group management device 11 may be configured as one communication network or multiple communication networks. These networks may be, for example, a communication network or communication path of any communication standard, such as the Internet, a public telephone line network, a wide area communication network for wireless mobile devices such as a so-called 4G line or 5G line, a WAN (Wide Area Network), a LAN (Local Area Network), a wireless communication network that communicates in accordance with the Bluetooth (registered trademark) standard, a communication path for short-distance wireless communication such as NFC (Near Field Communication), a communication path for infrared communication, a communication network for wired communication in accordance with standards such as HDMI (registered trademark) (High-Definition Multimedia Interface) or USB (Universal Serial Bus), etc.

[0020] A plurality of satellites 21 constitute a satellite group 31. In Fig. 1, satellites 21A and 21B constitute a first satellite group 31A, and satellites 21C and 21D constitute a second satellite group 31B. Note that, for simplicity, the example in Fig. 1 shows an example in which one satellite group 31 is constituted by two satellites 21, but the number of satellites 21 that constitute one satellite group 31 is not limited to two.

[0021] When the communication device 13 communicates with each satellite 21 constituting the satellite group 31, there are two methods: one is to communicate with each satellite 21 individually as in the first satellite group 31A in Fig. 1, and the other is to communicate with the communication device 13 only with one satellite 21C (hereinafter also referred to as representative satellite 21C) representing the satellite group 31 as in the second satellite group 31B, and the other satellites 21D communicate with the communication device 13 indirectly through inter-satellite communication with the representative satellite 21C. Which method is used to communicate with the earth station 15 (the communication device 13) may be predetermined by the satellite group 31, or may be appropriately selected depending on the content of the communication.

[0022] In the satellite image processing system 1 configured as above, the multiple satellites 21 constituting one satellite group 31 may be operated by an operation method called formation flight.

[0023] As shown in Fig. 2, formation flight is an operation method in which multiple satellites 21 constituting one satellite group 31 fly while maintaining their relative positional relationship within a narrow range of about several hundred meters to several kilometers, and the multiple satellites 21 operate in cooperation, and it is possible to provide services that cannot be realized by a single satellite. In Fig. 2, three satellites 21X to 21Z constitute one satellite group 31, and each of the satellites 21X to 21Z communicates with the ground station 15. In the uplink, a command or data is transmitted to a desired satellite 21 by specifying a group ID (satellite group ID) that is an identifier for identifying the satellite group 31, and an individual ID (satellite ID) that is an identifier for identifying each satellite 21 constituting the satellite group 31.

[0024] Formation flight has the advantage that functions can be shared not by a single satellite but by multiple satellites 21, making it possible to miniaturize each satellite 21. For example, in terms of imaging function, even if the performance (e.g., resolution) of the imaging device mounted on each satellite 21 is reduced, high resolution can be achieved by image synthesis of images captured by the multiple satellites 21.

[0025] For example, as shown in A of Fig. 3, two satellites 21E and 21F can simultaneously capture (simultaneously photograph) one area 52 from different imaging points (satellite positions). The results of capturing images of the same earth surface from different imaging points can be used to generate a digital elevation model (DEM) that indicates the height required for three-dimensional measurement. Also, a parallax image can be obtained from the captured images of the two satellites 21E and 21F, allowing three-dimensional measurement to be performed.

[0026] 3B, multiple satellites 21E and 21F can capture images of one area 52 at the same imaging point and angle with a time lag (differential photography). For example, if the satellites 21 move at 7 km per second and the distance between the satellites 21 flying in formation is 100 m, the difference in time between the satellites 21 is 1.4×10 -2 In this way, formation flights can capture images at short time intervals, making it possible to extract changes (displacements) in objects on the Earth, such as cars on the road or buoys on the sea, and to measure the speed of moving objects.

[0027] A constellation is a system that operates multiple satellites 21, but a constellation is "a system that deploys services uniformly, primarily around the globe, by launching a large number of satellites into a single or multiple orbital planes," and is a different concept from a formation flight.

[0028] FIG. 4 is a block diagram showing an example of the configuration of the satellite 21.

[0029] The satellite 21 includes a management unit 101, a bus 102, an imaging control unit 103, a thermal control unit 104, an attitude control system control unit 105, an orbit control system control unit 106, a propulsion system control unit 107, a sensor control unit 108, a power supply control unit 109, and a communication control unit 110. The satellite 21 also includes an imaging device 111, a cooling device 112, an attitude control device 113, a propulsion device 114, a group of sensors 115, a battery 116, a solar panel 117, and a communication device 118. The management unit 101 and the control units of each device, that is, the imaging control unit 103, the thermal control unit 104, the attitude control system control unit 105, the orbit control system control unit 106, the propulsion system control unit 107, the sensor control unit 108, the power supply control unit 109, and the communication control unit 110, are connected via the bus 102.

[0030] The management unit 101 obtains the status of each device from the control unit of each device via the bus 102 and outputs operation commands to the control unit of each device, thereby controlling the operation of the entire satellite 21.

[0031] The imaging control unit 103 controls the operation of the imaging device 111 according to an operation command from the management unit 101. The imaging device 111 is composed of, for example, a camera module including an image sensor, and captures an image of an object. If the satellite 21 is a synthetic aperture radar (SAR) satellite, the imaging device 111 is composed of a radar device.

[0032] The thermal control unit 104 acquires the sensor values ​​of the temperature sensors included in the sensor group 115, monitors temperature changes inside the satellite 21, and controls the entire satellite 21 to be within a specified temperature range. Basically, the temperature change is controlled by the structure and material characteristics, but dynamic cooling may be performed using the cooling device 112 as necessary. The cooling device 112 performs cooling using a cryogen such as liquid helium, for example.

[0033] The attitude control system control unit 105 controls the attitude control device 113 according to an operation command from the management unit 101, thereby controlling the satellite 21 to face the target direction. For example, the attitude control system control unit 105 controls the antenna 14 to face the ground station 15, the solar panel 117 to face the sun, and the observation sensor such as the imaging device 111 to face the observation target direction. The attitude control device 113 is composed of, for example, a three-axis gyro, a wheel such as a control moment gyro, a magnetic torquer, etc. The attitude control system control unit 105 may use not only the attitude control device 113 but also the propulsion device 114 for attitude control purposes. When performing attitude control, the attitude control system control unit 105 acquires sensor values ​​of various sensors of the sensor group 115 as necessary. Examples of sensors used for attitude control purposes include a sun sensor, an earth sensor, a star sensor, a magnetic sensor, and a gyro.

[0034] The orbit control system control unit 106 maintains the orbit altitude and controls changes in the orbit. The orbit control system control unit 106 cooperates with the propulsion system control unit 107 and the propulsion device 114 to perform control.

[0035] The propulsion system control unit 107 controls the propulsion device 114 according to an operation command from the management unit 101. The propulsion device 114 is composed of, for example, a solid motor, an ion engine, an apogee engine, etc. The propulsion system control unit 107 acquires sensor values ​​from various sensors in the sensor group 115 as necessary, and operates the propulsion device 114 in cooperation with the attitude control device 113, thereby performing attitude control and attitude control of the satellite 21. If the satellite 21 is a small satellite, a chemical propulsion thruster or the like may not be mounted for the purpose of attitude control.

[0036] The sensor control unit 108 controls various sensors included in the sensor group 115, and supplies the sensor values ​​to the management unit 101 and other control units. The various sensors are sensors for monitoring the state inside the satellite 21, and are composed of, for example, a GPS receiver, a star tracker (attitude sensor), an acceleration sensor, a gyro sensor, a magnetic sensor, a temperature sensor, a sun sensor, an earth sensor, a star sensor, and the like.

[0037] The power supply control unit 109 controls the battery 116 and the solar panel 117. Under the control of the power supply control unit 109, the power generated by the solar panel 117 is stored in the battery 116. The power of the battery 116 may be directly distributed to each device in the satellite 21 or may be distributed via the bus 102.

[0038] The communication control unit 110 controls the communication device 118 according to an operation command from the management unit 101. The communication device 118 has an antenna, and communicates with the communication device 13 of the ground station 15 according to the control of the communication control unit 110. The communication device 118 can also communicate with other satellites 21 and relay satellites 22 that constitute the same satellite constellation 31. The communication control unit 110 and the communication device 118 may be configured to have separate systems for sending and receiving commands and telemetry, which are small amounts of data, and for sending and receiving mission-related data (imaging data, etc.), which are large amounts of data.

[0039] Each of the control units from the imaging control unit 103 to the communication control unit 110 may be further divided into two or more, any two or more may be integrated, or may be integrated with the management unit 101. Computational resources such as a CPU (Central Processing Unit) and memory are basically mounted on the management unit 101, but may also be mounted on each control unit. Each control unit may be implemented in a common hardware module.

[0040] The imaging devices 111 of the satellites 21 constituting one satellite group 31 may have the same performance or may have different performance.

[0041] For example, if the imaging devices 111 of the same model number are used as the imaging devices 111 mounted on each satellite 21 and the satellites 21 have the same performance, there are the following advantages. For example, images with the same performance can be acquired with a short time difference, and differences can be easily detected. Also, a high-precision (high-resolution) image can be generated by combining images captured in a shared manner. Also, since redundancy can be provided, a malfunction in one satellite can be tolerated.

[0042] On the other hand, if the imaging devices 111 mounted on each satellite 21 have different performance, they can be made to perform different roles, such as high-sensitivity monochrome imaging and low-sensitivity color imaging. Different performance includes not only cases where the mounted hardware configuration is different, but also cases where the mounted hardware configuration is the same but the performance is made different by performing different controls. For example, for image sensors of the same model number, one satellite 21 may acquire high-sensitivity low-resolution images by increasing the shutter speed, while the other satellite 21 may acquire low-sensitivity high-resolution images.

[0043] Examples of sharing the performance of the imaging devices 111 of the multiple satellites 21 may include control to make each device different in sensitivity / shutter speed, resolution, monochrome / color / polarized light, band (wavelength range), or a combination thereof. Also, the battery performance and communication performance may be different between the multiple satellites 21.

[0044] FIG. 5 is a block diagram showing an example of the configuration of the satellite constellation management device 11, the communication device 13, and the image analysis server 42.

[0045] The satellite group management device 11 includes a control unit 211 , a communication unit 212 , an operation unit 213 , and a display unit 214 .

[0046] The control unit 211 manages the multiple satellites 21 owned by the satellite operating company by executing a satellite management application program stored in a storage unit (not shown). For example, the control unit 211 uses related information acquired from the information providing server 41 as necessary to determine an operation plan for the multiple satellites 21, and issues instructions for attitude control and imaging to each satellite 21 via the communication device 13. The control unit 211 also generates metadata for captured images transmitted from the satellites 21 via the communication device 13, and performs processing to add the metadata to the captured images.

[0047] The communication unit 212 performs predetermined communication with the communication device 13 via the network 12 in accordance with instructions from the control unit 211, and also performs predetermined communication with the image analysis server .

[0048] The operation unit 213 is composed of, for example, a keyboard, a mouse, a touch panel, and the like, and receives commands and data input based on operations by a user (operator), and supplies them to the control unit 211.

[0049] The display unit 214 is composed of, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display, and displays the screen of the satellite management application program, as well as images captured by the satellite 21 and processed images that have been subjected to predetermined image processing on the captured images.

[0050] The communication device 13 includes a satellite communication unit 221 , a control unit 222 , and a communication unit 223 .

[0051] The satellite communication unit 221 communicates with each of the target satellites 21 in the satellite group 31 via the antenna 14 under the control of the control unit 222 .

[0052] The control unit 222, under the control of the satellite group management device 11, causes the satellite communication unit 221 to communicate with the satellite 21. In addition, the control unit 222 transmits data such as captured images acquired from the satellite 21 to the satellite group management device 11 via the communication unit 223.

[0053] The communication unit 223 performs predetermined communication with the satellite group management device 11 based on the control of the control unit 222 .

[0054] The image analysis server 42 includes a control unit 231 , a communication unit 232 , an operation unit 233 , and a display unit 234 .

[0055] The control unit 231 executes an image analysis application program stored in a storage unit (not shown) to perform predetermined image processing on the captured images supplied from the satellite group management device 11, such as metadata generation processing for adding predetermined metadata to the captured images, correction processing such as distortion correction of the captured images, and image synthesis processing such as color synthesis processing.

[0056] The communication unit 232 performs predetermined communication with the satellite group management device 11 or other devices under the control of the control unit 231. For example, the communication unit 232 receives images captured by the satellites 21 from the satellite group management device 11 and supplies them to the control unit 231, and transmits processed images after image processing to the satellite group management device 11.

[0057] The operation unit 233 is configured with, for example, a keyboard, a mouse, a touch panel, and the like, and receives commands and data input based on operations by a user (operator), and supplies them to the control unit 231.

[0058] The display unit 214 is configured, for example, with an LCD or an organic EL display, and displays the screen of an image analysis application program, images before or after image processing, and the like.

[0059] The satellite 21 and other devices constituting the satellite image processing system 1 are configured as described above.

[0060] The satellite group management device 11 selects the most suitable communication device 13 from among multiple communication devices 13 according to the orbit of the satellite 21 with which to communicate, and causes the selected communication device 13 to transmit predetermined commands such as an image capture instruction, and receives data such as captured images via the communication device 13. Since the satellite group management device 11 performs predetermined communications together with the communication device 13 arbitrarily selected according to the target satellite 21, in the following explanation, the satellite group management device 11 and the communication device 13 will be collectively referred to as the management system.

[0061] <2. Single camera imaging sequence> Next, with reference to the flowchart in FIG. 6, an imaging sequence focusing on a specific satellite 21 of the satellite group 31 performing a formation flight will be described.

[0062] First, in step S11, the management system determines requirements for imaging by the satellite 21 based on customer requirements.

[0063] Specifically, the management system determines, as imaging requirements, imaging date and time, imaging location, environmental conditions for imaging, camera settings, etc. The environmental conditions for imaging include, for example, weather conditions such as cloud cover at the imaging date and time, and the camera settings include, for example, resolution, zoom, shutter speed, sensitivity, aperture, etc.

[0064] In step S12, the management system determines which satellites 21 and ground stations 15 (communication devices 13 thereof) meet the imaging requirements.

[0065] Specifically, the management system selects a satellite 21 that meets the determined imaging requirements. For example, the satellite 21 is selected by determining whether the satellite will pass above the imaging target position on the determined imaging date and time, whether the imaging target position is within the observation width of the satellite 21, whether the imaging device 111 mounted on the satellite 21 meets the requirements for resolution and the determined camera setting values, etc. Then, a ground station 15 suitable for communicating with the selected satellite 21 is determined.

[0066] Furthermore, the management system can select a satellite 21 in consideration of the estimated remaining battery power of the satellite 21 at the imaging date and time, the power consumption for imaging, etc. For example, if the selected satellite 21 is scheduled to perform another imaging immediately before the imaging date and time, it is assumed that the next imaging cannot be performed because power will be consumed by the imaging, attitude control, data communication, thermal control, etc. associated with the imaging, and therefore the satellite 21 is selected by setting the priority of the satellite 21 according to the estimated remaining battery power and the power consumption for imaging.

[0067] In step S13, the management system directs the antenna 14 of the selected ground station 15 to the assumed orbit. The satellite constellation management device 11 transmits orbit information of the selected satellite 21 to the communication device 13, and the communication device 13 directs the antenna 14 to the assumed orbit.

[0068] In step S14, the management system transmits (uplinks) an imaging instruction to the selected satellite 21. That is, the satellite group management device 11 transmits a command to transmit an imaging instruction to the communication device 13 of the selected ground station 15, and the communication device 13, having received the command, transmits the imaging instruction to the selected satellite 21 via the antenna 14. The imaging instruction includes the imaging date and time, the imaging location, the camera setting values, and the like.

[0069] In step S31, the satellite 21 receives an image capturing instruction from the ground station 15, and in step S32, transmits a reception completion to the ground station 15.

[0070] In step S15, the management system receives a reception completion notification from the satellite 21 and stops sending the imaging instruction. The transmission of the imaging instruction from the ground station 15 is repeatedly executed until a reception completion response is received from the satellite 21.

[0071] In step S33, the satellite 21 performs imaging preparation processing based on the received imaging instruction. For example, the satellite 21 controls the attitude of the satellite 21 or the direction of the imaging device 111 so that the imaging device 111 faces the imaging target position as necessary (pointing). Also, for example, the imaging control unit 103 sets the zoom, shutter speed, sensitivity, aperture, etc. of the image sensor. Furthermore, the power supply control unit 109 performs charging in advance so that sufficient power is available on the imaging date and time.

[0072] When the imaging date and time designated in the imaging instruction arrives, the satellite 21 captures an image of the imaging target position in step S34.

[0073] In step S35, the satellite 21 generates metadata, which is information associated with the captured image obtained as a result of imaging, and adds it to the captured image. Details of the metadata will be described later, but for example, information such as a group ID for identifying the satellite group 31, an individual ID for identifying each satellite 21, the position of the image target (position of the subject), and the image capture time can be generated as metadata.

[0074] In step S36, the satellite 21 transmits (downlinks) the captured image with the metadata added to the ground station 15. The downlink may be performed immediately after the captured image and metadata are generated, or may be performed when the satellite 21 arrives within a predetermined range of a predetermined ground station 15. The captured image may also be transmitted via a relay satellite 22.

[0075] In step S16, the management system receives the captured images from the satellites 21. Specifically, the communication device 13 receives the captured images via the antenna 14 and supplies them to the satellite group management device 11.

[0076] In step S17, the satellite group management device 11 analyzes the metadata of the captured image. At this time, the satellite group management device 11 may generate new metadata based on the analysis result and add this. For example, the satellite group management device 11 calculates the satellite position at the time of capturing the image based on the group ID and individual ID of the captured image and the orbital information of the satellite 21, and adds it as metadata.

[0077] In step S18, the satellite group management device 11 performs predetermined image processing on the captured images captured by the satellites 21. The satellite group management device 11 performs, for example, correction processing such as distortion correction, image synthesis processing such as color synthesis, etc. Details of the image processing will be described later.

[0078] In step S19, the satellite group management device 11 executes a distribution management process for the captured images and the processed images, and stores them in a predetermined storage unit. The details of the distribution management process will be described later.

[0079] This completes a series of sequences in which one satellite 21 takes images. Image processing by image analysis server 42 can be performed appropriately as necessary, and can be shared with image processing by satellite group management device 11 or can be performed instead of satellite group management device 11. Distribution management processing may also be performed by image analysis server 42.

[0080] In the above example, the metadata is added to the captured image and then transmitted, but the metadata may be transmitted as a stream separate from the captured image. In this case, only the metadata may be transmitted prior to the captured image.

[0081] <3. Imaging preparation processing> Incidentally, since resources are limited, particularly in the case of a small satellite 21, attention must be paid to the remaining battery power, and imaging control in response to this is important.

[0082] Fig. 7 is a detailed flowchart of the imaging preparation process in step S33 in Fig. 6. Here, it is assumed that the imaging instruction received in step S31 prior to step S33 instructs imaging at imaging time t1.

[0083] In the imaging preparation process, first, in step S51, the management unit 101 of the satellite 21 estimates the remaining battery capacity at imaging time t1. Specifically, the management unit 101 estimates the remaining battery capacity at imaging time t1 from (an estimated value of) the charge amount accumulated by solar power generation up to imaging time t1, relative to the current remaining battery capacity.

[0084] In step S52, the management unit 101 determines whether or not the remaining battery charge is sufficient based on the estimated remaining battery charge.

[0085] Specifically, the management unit 101 determines whether the estimated remaining battery charge is sufficient based on power consumption factors related to imaging and power consumption factors other than imaging. Power consumption factors related to imaging include imaging processing of the imaging device 111, attitude control (pointing) of the satellite 21, and thermal control associated therewith. In the imaging processing of the imaging device 111, the number of images to be captured and the level of accuracy (resolution, shutter speed, necessity of zoom, etc.) to be captured at imaging time t1 are taken into consideration. Attitude control of the satellite 21 includes changing the attitude of the satellite itself and changing the attitude of the antenna. In addition, if the camera module itself as the imaging device 111 can change its attitude in the imaging direction, changing the attitude of the camera module is also included in the attitude control of the satellite 21. Power consumption factors other than imaging include communication (uplink and downlink) performed until imaging time t1.

[0086] For example, as shown in A of FIG. 8, assuming that the battery 116 is always maintained at 70% of its full charge, the current battery remaining charge is 90%, the charge until time t1 is 5%, the power consumption due to the imaging process at time t1 is 3%, the power consumption due to attitude control is 10%, and the power consumption due to communication until the imaging time t1 is 2%, then 90% + 5% - 3% - 10% - 2% = 80%, and since 70% charge can be secured even after imaging at time t1, the satellite 21 can determine that there is sufficient battery remaining.

[0087] The management unit 101 may also take into consideration the imaging performed at the timing following the imaging time t1 and determine whether the remaining battery charge is sufficient based on the battery charge that should remain after the imaging time t1.

[0088] For example, as shown in B of FIG. 8, assuming that imaging is scheduled at time t2 following imaging time t1, and the charge amount from time t1 to time t2 is 2%, the power consumption due to the imaging process at time t2 is 3%, the power consumption due to attitude control is 10%, and the power consumption due to communication performed up to the imaging time t2 is 2%, then a remaining battery charge of 83% is required after imaging at time t1, and therefore the estimated remaining battery charge of 80% at the imaging time t1 is determined to be insufficient.

[0089] In the above example, the power consumption related to imaging has been mainly described, but other power consumption, such as power consumption due to thermal control associated with attitude control and periodic communication, is also taken into consideration.

[0090] In the above manner, it is determined whether there is sufficient battery power remaining, and if it is determined in step S52 of Fig. 7 that there is not sufficient battery power remaining, the process proceeds to step S53, where the satellite 21 determines whether it is possible to change the downlink timing expected by the imaging time t1. By changing the downlink timing, it is possible to save the amount of power required for it.

[0091] If it is determined in step S53 that the downlink timing cannot be changed, the process of step S53 is skipped and the process proceeds to step S55.

[0092] On the other hand, if it is determined in step S53 that the downlink timing can be changed, the process proceeds to step S54, where the management unit 101 changes the downlink timing and determines whether there is sufficient battery remaining power as a result. If it is determined in step S54 that there is not sufficient battery remaining power, the process proceeds to step S55. On the other hand, if it is determined in step S54 that there is sufficient battery remaining power, the process proceeds to step S57.

[0093] In step S55, the management unit 101 changes the accuracy of the attitude control. For example, the attitude control involves applying a moment toward the target attitude using two types of devices, a wheel and an ion engine, and then repeatedly applying a reverse moment when the target attitude is passed. When the sway speed becomes equal to or lower than a certain level, it is determined that the attitude has been changed to the target attitude. To change the accuracy of the attitude control, the management unit 101 changes, for example, the range of the sway speed at which it is determined that the target attitude has been achieved. By changing the range of the sway speed in the direction of increasing it and reducing the amount of control of the attitude control, it is possible to save power consumption.

[0094] In step S56, the management unit 101 changes the imaging conditions according to the accuracy of the attitude control. If the range of the shaking speed becomes large, the attitude of the satellite 21 will not be stable and wobble will occur, which may cause subject blurring. Also, it is possible that sufficient zooming will not be possible due to insufficient pointing. Therefore, the management unit 101 compensates for the adverse effects caused by the reduction in the control amount of the attitude control by changing the imaging conditions.

[0095] The management unit 101 changes the imaging conditions, for example, as follows.

[0096] The management unit 101 deals with subject blur by increasing the shutter speed of the image sensor. In addition, since the captured image becomes darker when the shutter speed is increased, the management unit 101 may also perform control to increase the sensitivity (gain).

[0097] For example, the management unit 101 can reduce the resolution of the captured image in order to improve the sensitivity per unit pixel. This can improve the shutter speed, reduce the influence of a decrease in the accuracy of attitude control, and reduce the amount of data during downlink. The management unit 101 also selects a setting value that does not perform optical zoom. This can increase the tolerance for image blurring (shaking).

[0098] Furthermore, if the camera module is equipped with a mechanical blur correction mechanism (spatial blur correction), the mechanical blur correction mechanism may be used instead of lowering the accuracy of the attitude control.

[0099] Furthermore, the management unit 101 may set the imaging so as to capture multiple consecutive images instead of reducing the resolution of the captured image. The ground station 15 can compensate for the reduction in resolution of the captured image by generating a high-resolution captured image by synthesizing the continuously captured images and transmitting (downlinking) the generated high-resolution captured image. Note that the satellite group management device 11 or the image analysis server 42 may generate a high-resolution image by image synthesis after downlinking. The satellite group management device 11 or the image analysis server 42 can also synthesize the image with past captured images such as a base image or captured images captured by other satellites 21.

[0100] After step S56, or if it is determined in step S52 or step S54 that there is a sufficient remaining battery charge, the process proceeds to step S57.

[0101] In step S57, the management unit 101 controls the attitude of the satellite 21 or the imaging device 111 (performs pointing) in accordance with the attitude control settings determined in the process of step S55.

[0102] In step S58, the management unit 101 sets the imaging conditions determined in the process of step S56.

[0103] With the above, the imaging preparation process in step S33 in FIG. 6 is completed, and when the imaging date and time designated by the imaging instruction arrives, the process in step S in FIG. 6 is performed, that is, imaging of the imaging target position is performed.

[0104] According to the image capture preparation process, when the battery level is low, the stability accuracy of attitude control, which has a large impact on power consumption, is lowered, and the image capture conditions and subsequent image processing are changed, thereby reducing battery consumption while ensuring the quality of the captured image.

[0105] <4. Formation flight flow chart> Next, a formation flight performed by a plurality of satellites 21 constituting one satellite group 31 will be described.

[0106] FIG. 9 is a flowchart of the satellite image processing system 1 in which one satellite group 31 performs a formation flight.

[0107] First, the management system and each satellite 21 of the satellite group 31 performing the formation flight perform the relative position confirmation process of steps S101, S121, S122, and S102. That is, in step S101, the management system inquires of each satellite 21 of the satellite group 31 performing the formation flight about the relative position. In step S121, each satellite 21 constituting the satellite group 31 performs a process of confirming the relative position in response to the inquiry from the management system. Then, in step S122, each satellite 21 transmits the relative position, and in step S102, the management system receives the relative position from each satellite 21. Here, the relative position is the arrangement order of each satellite 21 constituting the satellite group 31 and the distance between the satellites. The arrangement order of each satellite 21 is, for example, the order in which the satellite 21 moves (number 1). This relative position confirmation process may be performed every time imaging is performed, or may be performed periodically, for example, once a day or once a week.

[0108] The management system has orbital information of the satellite constellation 31 obtained from NORAD, an external organization, but the orbital information of each satellite 21 constituting the satellite constellation 31 may not be distinguishable. Or, even if the individual orbital information can be distinguished by observation from the ground, the order of the aircraft may not be distinguishable. In a formation flight, the satellites 21 may be positioned to such an extent that orbital information cannot be individually assigned, and it is not possible to distinguish the number of a certain satellite from the beginning of the satellite constellation 31. For this reason, it is necessary to determine the relative positional relationship.

[0109] Methods for controlling the relative position are broadly divided into two types: open loop and closed loop.

[0110] The open loop method is a method in which there is no communication between the satellites that make up the satellite group 31, and the relative positions are controlled by instructions from the ground. Errors in the distance between satellites are likely to occur.

[0111] In contrast, the closed-loop method is a method of controlling relative positions by communicating between satellites that make up the satellite group 31. The accuracy of relative positions in the closed-loop method is higher than that in the open-loop method. There are two types of closed-loop methods: a centralized type and a decentralized type. The centralized type has a leader satellite 21, and the other satellites 21 follow the leader satellite, and a leader satellite issues instructions to the other satellites 21. The decentralized type is a type in which each satellite 21 that makes up the satellite group 31 autonomously communicates with the other satellites 21 in the vicinity and controls its own position.

[0112] In the open-loop method, the process of confirming the relative positions in step S121 may be, for example, a method in which each satellite 21 simultaneously captures an image of a predetermined point on the ground, and the satellite group management device 11 on the ground side confirms the arrangement order of each satellite 21 based on the captured image and the attitude (pointing angle) information of the satellite 21. In another example, each satellite 21 simultaneously communicates with a predetermined point on the ground, and the communication device 13 on the ground side confirms the arrangement order from the radio waves at that time. The communication for confirming the arrangement order may be a downlink of a predetermined captured image or a signal for calibration. On the other hand, in the closed-loop method, each satellite 21 executes a process of measuring the relative position, and the measurement result is downlinked. Methods for each satellite 21 to measure the relative position include a method of measuring the position (direction) by communicating with each other, a method of the satellite 21 emitting a laser and measuring the distance from the reflected light, and the like.

[0113] In the closed loop and open loop methods, only the sequence of the satellites 21 may be detected, and the distances between the satellites may be calculated by observation from the ground.

[0114] In step S103, the management system calculates the imaging conditions for each satellite 21 based on the relative position of each satellite 21. The imaging conditions here include not only the settings of the image sensor, but also attitude control of the satellite 21 during imaging and the timing of imaging. For example, when performing three-dimensional measurement on the ground, imaging conditions are calculated for making each satellite 21 assume an attitude such that the imaging target position is the same, with the inter-satellite distance as the baseline length. When imaging with a time difference (differential imaging) is performed by multiple satellites 21 constituting the satellite group 31, the timing (imaging position) at which the preceding satellite 21 and the following satellite 21 perform imaging and the attitude at the time of imaging are calculated. The timing at which each satellite 21 performs imaging is calculated based on the inter-satellite distance.

[0115] In step S104, the management system transmits an imaging instruction to each satellite 21 based on the calculated imaging conditions. The imaging instruction is transmitted (multicast) to all satellites 21 in the satellite group 31, but each satellite 21 can select an instruction addressed to itself based on an individual ID as destination information included in the imaging instruction.

[0116] The satellite 21 receives an image capturing instruction from the ground station 15 in step S123, performs image capturing preparation processing in step S124, and captures images in step S125. Furthermore, the satellite 21 generates metadata and adds it to the captured image in step S126, and transmits (downlinks) the captured image with the added metadata to the ground station 15 in step S127.

[0117] The processing of steps S123 to S127 is basically the same as the processing of steps S31 to S36 performed by each satellite 21 described with reference to Fig. 6. Note that, with regard to transmission of the captured images in step S127, each satellite 21 may transmit its own captured images individually, or the captured images may be collected by a leader satellite through inter-satellite communication and transmitted collectively by the leader satellite.

[0118] In step S105, the management system receives captured images from each satellite 21, and in step S106, analyzes metadata of the captured images. Furthermore, in step S107, the management system performs a predetermined image processing on the captured images, and in step S108, executes a distribution management process for the captured images and the processed images, and stores them in a predetermined storage unit.

[0119] The processing of steps S105 to S108 is basically the same as the processing of steps S16 to S19 performed by the management system described with reference to Fig. 6. However, the image processing of step S107 is not limited to image processing of an image obtained by one satellite 21, but can also perform image processing using a plurality of images taken by a plurality of satellites 21 of the satellite group 31 in cooperation with each other.

[0120] <5. Example of image processing> An example of the image processing executed by the satellite group management device 11 or the image analysis server 42 in step S18 in FIG. 6 or step S107 in FIG. 9 will be described.

[0121] For each image captured by each satellite 21, the satellite group management device 11 or the image analysis server 42 can perform the following image processing.

[0122] (1) Metadata generation Metadata can be generated based on information transmitted from the satellite 21 and information about the satellite 21 that has taken the image. For example, information on the latitude and longitude of the image capture target position, and information on the attitude control and acceleration of the satellite 21 when taking the image can be generated as metadata. (2) Correction of captured images Correction processes that can be performed include radiometric correction related to sensitivity characteristics, geometric correction of the orbital position and attitude error of the satellite 21, ortho-correction to correct geometric distortion caused by differences in elevation of the terrain, and map projection to project an image onto a map projection surface. (3) Color composition processing It is possible to perform color synthesis processes such as pan-sharpening, true color synthesis, false color synthesis, natural color synthesis, SAR image synthesis, and processing to add color to the captured images for each band. (4) Other image synthesis It is also possible to combine images captured by the satellite 21 in the past, images captured by other satellites 21, or some other base image, combine images captured in different bands, or combine images with map information. (5) Information Extraction Using different bands such as R (Red) and IR (Infrared), it is possible to calculate vegetation detection information such as NDVI (Normalized Difference Vegetation Index) and water detection information such as NDWI (Normalized Difference Water Index). It is possible to perform highlight processing of specific subjects such as vehicles, moving objects, and schools of fish, and to extract information on specific bands and changes from the previous image capture.

[0123] In particular, when using multiple images captured by multiple satellites 21 performing formation flight, the satellite group management device 11 or the image analysis server 42 can more effectively perform image processing such as the following:

[0124] (1) High-resolution or high-quality processing By superimposing multiple captured images, a captured image with improved resolution can be generated. Also, a pan-sharpened image can be generated by combining a monochrome image and a color image, or a high-resolution captured image can be generated by combining captured images with different imaging conditions, such as different dynamic ranges, shutter speeds, different bands (wavelength bands), and different resolutions. (2) Division of Functions Different bands such as R (Red) and IR (Infrared) can be used to calculate indices such as NDVI (Normalized Difference Vegetation Index). (3) 3D measurement Parallax images can provide three-dimensional information. In addition, three-dimensional information can improve the accuracy of object recognition on the ground. For example, it can be used to determine whether an object is a vehicle or not (even if the resolution of the image does not immediately indicate that it is a vehicle, if it is determined that the object on the road is a three-dimensional object rather than a pattern, it can be assumed that it is a vehicle). (4) Differential measurement Using multiple captured images taken at the same position with a time difference, it is possible to extract the change between the first time and the second time. Also, imaging may be performed such that only the changed object is extracted and colored. Also, for example, using multiple captured images, it is possible to calculate the moving speed of a ship or a vehicle, or to calculate the wind speed from the movement of clouds, etc. (5) Other image synthesis It is also possible to combine images captured in the past, images captured by other satellites 21, images captured in different bands, and images with map information.

[0125] The satellite group management device 11 and image analysis server 42 as an image processing device perform the above-mentioned image processing based on satellite identification information that identifies the satellite and is associated as metadata with the captured image captured by the satellite 21. In other words, by associating the satellite identification information as metadata with the captured image, it becomes possible to process multiple images using the relative positional relationship of the multiple satellites 21 in the formation flight. The satellite identification information includes at least a group ID that identifies the satellite group 31, an individual ID that identifies each satellite 21 that constitutes the satellite group 31, and relative position information of each satellite 21 performing the formation flight.

[0126] Although the image processing using a plurality of captured images taken by a formation flight has been described, the above-mentioned image processing may be performed on a plurality of captured images taken by a constellation, not a formation flight. For example, image processing such as (1) high-resolution or high-quality processing, (3) three-dimensional measurement, and (5) other image synthesis may be performed on a plurality of captured images taken by a constellation.

[0127] (Image format) The processed image after image processing and the captured image are stored in a storage unit using, for example, the following image format, and are provided to a customer or the like. (1) CEOS CEOS is a format standardized by the Committee on Earth Observation Satellites. There are two types of CEOS: "CEOS-BSQ", in which the files are divided by band, and "CEOS-BIL", in which multiple bands are multiplexed. (2) HDF This format was developed by the National Center for Supercomputing Applications (NCSA) at the University of Illinois. Multiple bands are packed into one file to facilitate data interchange among various computer environments. (3) Geo TIFF This is a format that adds remote sensing information to TIFF (Tagged Image File Format). Since it is in TIFF format, it can be opened with a general image viewer. (4)JPEG2000 An image format standardized by the Joint Photographic Experts Group. JPEG 2000 does not simply increase the compression rate, but also incorporates techniques for improving images in the area of ​​interest and copyright protection techniques such as digital watermarking.

[0128] Methods of presenting processed images and captured images include (1) a method of providing images in a viewable manner, and (2) a method of presenting only information based on analysis of the images.

[0129] Furthermore, (1) methods of providing images in a viewable manner include (1A) providing (transmitting) the image itself, (1B) permitting access to a platform such as a data server so that users can view images from the data on the platform, and (1C) providing users with dedicated software for viewing images so that users can view images only on that dedicated software.

[0130] (2) A method of presenting only information based on image analysis is, for example, a method of presenting the number of vehicles or moving objects per hour or the area of ​​a school of fish obtained by performing the information extraction process described above.

[0131] <6. Metadata details> FIG. 10 shows an example of information that is added as metadata to a captured image or a processed image.

[0132] Depending on the type of information, the information added as metadata can be information that can be added by satellite 21, information that can be added by satellite constellation management device 11, or information that can be added by the image analysis server 42 of the analysis company, but in Fig. 10, each piece of information is arranged in a table format, and a circle (○) is marked on the device that can add each piece of information. Note that, if the satellite constellation management device 11 also has an image processing function, it goes without saying that the information that can be added by the image analysis server 42 can also be added by the satellite constellation management device 11 itself.

[0133] As the metadata, for example, information for identifying a satellite (satellite identification information) can be added. The information for identifying a satellite can include, for example, a group ID for identifying the satellite group 31, an individual ID for identifying each satellite 21, relative position information of each satellite 21 constituting the satellite group 31 performing a formation flight, angle information of itself (satellite 21) at the time of image capture, and satellite type. The relative position information includes, for example, information such as the order of the multiple satellites 21 constituting the satellite group 31 and the distance between the satellites. The relative position information may be information that serves as material for estimating the relative position. The angle information of itself at the time of image capture indicates, for example, the angle of itself with respect to the Earth's surface at the time of image capture. The satellite type includes, for example, whether it is an optical satellite or a SAR satellite, and a classification according to the use, size, etc. of the satellite.

[0134] In addition, the information for identifying the satellite may include, for example, orbit information of the satellite 21 in the TLE format (TLE information), position information from GPS signals (GPS information), orbital position and orbital altitude information calculated from at least one of the TLE information and the GPS information, velocity information of the satellite 21, and sensor information such as the satellite 21's earth sensor, sun sensor, star tracker, etc.

[0135] Information about the image content can be added to the metadata, which can include, for example, image target position information indicating the location on the earth that was the image target, image capture conditions such as resolution, zoom, shutter speed, sensitivity, aperture (F value), sensor type such as the model number of the image sensor, image capture time, satellite position at the time of image capture, weather information such as cloud cover and amount of sunlight, etc.

[0136] The imaging target position information is, for example, information on the latitude and longitude of the location on the earth that is the imaging target. The satellite position at the time of imaging is added on the ground side based on the orbital information of the satellite 21. The satellite position at the time of imaging may be the orbital information of the satellite 21 itself. In addition, in the above-mentioned imaging preparation process, the accuracy of the attitude control may be changed depending on the remaining battery power, so the satellite position at the time of imaging may further include information on the accuracy of the attitude control of the satellite 21 at the time of imaging and three-dimensional acceleration information indicating the movement of the satellite itself at the time of imaging. This information on attitude control can be used as a reference for processing in high-resolution processing of the captured image performed on the ground side.

[0137] Furthermore, information about the type of image can be added to the metadata, which can include band information and image processing information.

[0138] The band information includes wavelength information regarding the wavelength band, color information indicating whether it is RGB (True Color), IR (infrared), or monochrome, coloring information (False Color) indicating that specific objects such as plants have been colored, and analysis information indicating that the image represents the Normalized Difference Vegetation Index (NDVI) or Normalized Difference Water Index (NDWI).

[0139] The image processing information includes the processing time, processing level, and processing method of the image processing. The processing time indicates the time when the image processing is performed. The processing levels are divided into six stages from L0 to L5. L0 is a level indicating an uncorrected state where no correction processing has been performed, L1 is a level where radiometric correction regarding sensitivity characteristics has been performed, and L2 is a level where geometric correction such as the orbital position and attitude error of the satellite 21 has been performed. Other levels include a level where an image is projected onto a map projection surface, and a level where orthorectification that corrects geometric distortion has been performed. The processing method includes the name of the processing, such as pan-sharpening processing, true color synthesis processing, and SAR image synthesis processing. The processed image of the 3D measurement may be classified as an L image (image for the left eye) or an R image (image for the right eye).

[0140] Furthermore, related party information, which is information about related parties of the captured image or processed image, can be added to the metadata. Information about related parties includes, for example, information such as the owner of the satellite 21, the service operator who operates the satellite remote sensing service, and the right holder of the captured image or processed image. By adding related party information to the captured image or processed image as metadata, it is possible to manage related parties of the captured image or processed image by referring to or matching the related parties of the captured image or processed image, and it is possible to ensure the authenticity of the image.

[0141] <7. Details of distribution management processing> Next, the distribution management process of captured images or processed images executed by satellite group management device 11 or image analysis server 42 in step S19 in FIG. 6 and step S108 in FIG. 9 will be described.

[0142] The captured image and the processed image can be subjected to the following processing for managing the distribution of data. (1) Usage restriction processing It is possible to perform processing so that the captured image and the processed image cannot be downloaded or displayed without permission, or to perform processing so that the captured image and the processed image cannot be downloaded or displayed when certain conditions such as expiration date, number of copies, number of displays, etc. It is also possible to perform processing so that the captured image and the processed image cannot be subjected to secondary processing such as image synthesis. (2) Watermark It is possible to process captured images and processed images by inserting a watermark (digital watermark) indicating that the images have a copyright. It is also possible to process by inserting information that allows the leak route to be identified as a watermark.

[0143] By carrying out the above-mentioned distribution management process, it is possible to guarantee the authenticity of the images and prevent the leakage and inappropriate use of the captured images and processed images. In this case, a method of managing each data and the usage of the data using a blockchain may be adopted.

[0144] (Example of image protection processing) When a user requests privacy protection for captured images and processed images, or when an image shows an area (disclosure restricted area) or an area (prohibited area) where disclosure is restricted or prohibited by the laws of each country, such as a military facility or public facility, the satellite group management device 11 or the image analysis server 42 can apply a process to protect the image using a predetermined protection method. Whether or not the image is in a protected area can be determined using the captured object location information in the metadata.

[0145] Methods for protecting images include, for example, subjecting the images of the protected area to processing so that no one other than the end user or an authorized user can perform processing to increase the resolution more than necessary. Alternatively, the resolution of the images of the protected area may be reduced or blurred. Furthermore, updating of the images of the protected area may be stopped and the images may be replaced with older images or an image indicating protection may be superimposed.

[0146] In addition to protecting images before they are first provided to users, images can be protected after the fact when there is a request for privacy protection, when unauthorized distribution of images is detected, etc. When unauthorized distribution of images is detected, measures can be taken to delete the illegally leaked captured images and processed images.

[0147] By enabling satellite group management device 11 and image analysis server 42 to handle the image protection processing described above, it is possible to meet user requests for privacy protection and disclosure restrictions.

[0148] <8. Examples of Formation Flight Applications> An example of image analysis processing using images captured by a plurality of satellites 21 constituting a satellite constellation 31 by formation flight will be described below.

[0149] (1) Confirmation of crop germination through high-resolution imaging (agricultural remote sensing) Observations to confirm the germination of agricultural crops require a resolution of several centimeters. By combining images taken by multiple satellites in a formation flight, it is possible to achieve a resolution that exceeds the resolution of a single satellite, making it possible to detect germination. The satellite group 31 captures images of the same point on farmland as the imaging target position. Each satellite 21 may capture images simultaneously from different positions, or may capture images from the same position with a time difference. In order to orient the imaging target positions of each satellite 21 to the same point, it is necessary to know the satellite positions in advance. In the image synthesis process, it is not necessary to know which satellite 21 captured each image, but if it is possible to know which satellite 21 captured the image, the angle and time when the image was captured can be determined, enabling more efficient image synthesis. The format of the processed image after synthesis can be, for example, Geo TIFF format, and metadata can be added indicating that the image is a synthesis image obtained by a formation flight, as well as the imaging positions, imaging times, imaging conditions, etc. of the original captured images. The imaging position information can be that of one of the original captured images (representative captured image).

[0150] (2) Confirmation of crop growth status using 3D measurements (agricultural remote sensing) The growth status of crops is confirmed using indices such as NDVI, but it is also possible to obtain accurate height information using three-dimensional measurements. Each satellite 21 of the satellite group 31 simultaneously captures images of the same farmland as the imaging target position, and obtains parallax images. Relative position information of the satellites 21 is required to obtain the distance between the satellites, which is the baseline length. This relative position information may be obtained simultaneously with the downlink of the captured images, rather than in advance. In the image synthesis process, it is not necessary to know which satellite 21 captured each image, but if it is possible to know which satellite 21 captured the image, the angle and time when the image was captured can be determined, enabling more efficient image synthesis. The processed image after synthesis can use, for example, a format of a three-dimensional image consisting of a pair of an L image and an R image, and metadata can be added to indicate that it is a synthetic image obtained by a formation flight, as well as the imaging positions, imaging times, and imaging conditions of each of the original captured images. The imaging position information can be that of one of the original captured images (representative captured image). In addition to the three-dimensional measurement information, vegetation indices such as NDVI and other information can be further added.

[0151] (3) Other agricultural remote sensing For example, it is possible to obtain accurate height information for checking the level of farmland after it has been cultivated by using three-dimensional measurement.

[0152] (4) Detecting the movement of schools of fish (ocean observation remote sensing) It is possible to detect schools of fish and obtain information on their movement direction and speed. The satellite group 31 captures images from the same point in the ocean as the imaging target position. Each satellite 21 captures images from the same position with a time difference. In order to orient the imaging target positions of each satellite 21 to the same point, it is necessary to know the satellite positions in advance. In particular, when imaging the ocean where there is no reference target as the imaging target position, it is necessary to precisely align the captured images of each satellite 21, so it is important to know the relative positions and movement speed information of each satellite 21 in advance. In the analysis process of the captured images, the images captured by each satellite 21 are aligned based on the image capturing position (including angle information) and image capturing time, and the fish schools are compared. Through the comparison process, the movement speed of the fish schools can be calculated from the time difference between the image capturing times of two or more satellites 21 and the movement distance of the fish schools. The image presented as the analysis processing image may be, for example, an image in which information indicating the movement direction and speed of the school of fish is superimposed on a base image of the school of fish (image captured by a specific satellite 21). Various information about the base image is added to the metadata. As a result of the analysis process, information explaining the calculation method used to calculate the movement direction and speed of the school of fish may be presented, such as multiple images capturing the school of fish, the time they were captured, and the position of the school of fish. (5) Other ocean observation remote sensing For example, information on the direction and speed of ship movement, as well as observation information on ocean currents, can be obtained.

[0153] (6) Vehicle counting (economic indicators estimation) Economic indicators (economic trends and sales forecasts for specific stores) are calculated by investigating the number of vehicles in parking lots and the number of vehicles traveling on roads. By synthesizing images taken by multiple satellites during formation flights, high-resolution images can be generated, making it possible to more accurately detect the number of vehicles and the number of vehicles traveling. The satellite group 31 simultaneously captures images of the same target location. In order to orient the target location of each satellite 21 to the same location, it is necessary to know the satellite positions in advance. By capturing multiple images simultaneously, it is possible to increase the resolution of the images and obtain three-dimensional information based on parallax images. In the image synthesis process, it is not necessary to know which satellite 21 captured each image, but if it is possible to know which satellite 21 captured the image, the angle and time of capture can be determined, enabling more efficient image synthesis. When synthesizing two or more captured images, a reference object may be extracted from a road or building in the image, and the two or more images may be aligned based on this. The reference object may be selected based on height information. In the image analysis process, the number of vehicles and the number of vehicles traveling are calculated based on the high-resolution captured image. The number of vehicles and the number of vehicles traveling may be calculated efficiently by increasing the resolution of only a specific area in the captured image. If it is not possible to determine whether or not an object is a vehicle based on a two-dimensional image, the determination may be made based on three-dimensional information including height. The image presented as the analysis processing image may be, for example, a base captured image (image captured by a specific satellite 21) with detection target areas and count targets (vehicles and people) colored in different colors and with the count numbers superimposed. Various information about the base image is added to the metadata. As a result of the analysis process, information such as the image capturing conditions and the calculation method of the detected object may be presented to the user. Although the above-described example is an example of increasing the resolution by capturing images simultaneously, the vehicle movement speed may be measured based on images captured with a time lag, and traffic volume information before and after the capture time may be estimated and presented. (7)Other By synthesizing images captured by multiple satellites during a formation flight, it is possible to obtain 3D information based on parallax images and create 3D maps of construction sites and mansions. (8) Variations A constellation of formation flights may also be used. In other words, by launching a group of satellites 31 performing formation flights into a single or multiple orbital planes, it is possible to perform operations that provide services uniformly mainly around the globe. Images captured by the formation flight may be combined with images captured by other satellites. For example, image processing may be performed to superimpose moving object information obtained during the formation flight onto a high-resolution image captured by a geostationary satellite.

[0154] <9. Second embodiment of satellite image processing system> FIG. 11 shows an example of the configuration of a second embodiment of a satellite image processing system to which the present technology is applied.

[0155] In the first embodiment described above, the group of satellites 31 performing formation flight is configured to perform simultaneous imaging or time-difference imaging at imaging points and imaging times designated in advance based on the orbital information, etc., of the satellites 21. Therefore, for example, it is not possible to detect a specific event that occurs on the ground and perform real-time imaging at the time the event occurs.

[0156] In the second embodiment described below, a configuration will be described in which one or more satellites 21 perform real-time imaging in response to an event occurring on the ground. When a satellite group 31 consisting of multiple satellites 21 performs real-time imaging in response to an event occurring on the ground, the satellite group 31 may be operated in either a constellation or formation flight.

[0157] As shown in Fig. 11, a plurality of transmitting devices 251 equipped with sensors for detecting a predetermined event on the ground are newly added to the configuration of the satellite image processing system 1 of the second embodiment. In the example of Fig. 11, four transmitting devices 251A to 251D are installed in the event detection area 250, but the number of transmitting devices 251 is arbitrary. Note that the three satellites 21X to 21Z of the second embodiment shown in Fig. 11 may be operated in either a constellation or formation flight. Also, each of the three satellites 21X to 21Z may be a satellite 21 operated independently.

[0158] Each of the four transmitting devices 251A to 251D detects an event in a divided event detection area 250. The sector-shaped area indicated by the dashed line in Fig. 11 indicates the event detection range of one transmitting device 251. The event detection area 250 is, for example, farmland, and a sensor included in the transmitting device 251 monitors the temperature of the farmland and the growth status of the crops.

[0159] The transmitting device 251 detects a predetermined event in the event detection area 250, and transmits an imaging instruction to one or more satellites 21. In response to the imaging instruction transmitted from the transmitting device 251, the satellites 21X to 21Z capture images of the area in which the event has occurred.

[0160] FIG. 12 is a block diagram showing an example of the configuration of the transmission device 251. As shown in FIG.

[0161] The transmitting device 251 is composed of a transmitting section 271 , a control section 272 , a sensor section 273 , and a power supply section 274 .

[0162] The transmitting unit 271 , under the control of the control unit 272 , transmits an image capturing instruction to a satellite 21 passing near the transmitting device 251 .

[0163] The transmission unit 271 is, for example, omnidirectional and can transmit image capture instructions to all satellites 21 passing within a certain range of the transmission device 251. The transmission unit 271 is, for example, capable of long-distance communication of 100 km or more with a high-speed moving object moving at 100 km / h, and is configured of a low-power consumption communication device.

[0164] The transmitting unit 271 may be directional. In this case, the transmitting unit 271 directs an antenna (not shown) to a satellite 21 passing near the transmitting device 251 based on the orbital information of the satellite 21, and transmits an image capturing instruction to the target satellite 21. The orbital information of the satellite 21 is stored in advance.

[0165] The control unit 272 controls the overall operation of the transmission device 251. When a predetermined event is detected by the sensor unit 273, the control unit 272 controls the transmission unit 271 to transmit an image capturing instruction toward the satellite 21.

[0166] The sensor unit 273 is composed of one or more types of predetermined sensors depending on the purpose of event detection. For example, the sensor unit 273 is composed of an odor sensor, an air pressure sensor, a temperature sensor, etc. Furthermore, for example, the sensor unit 273 may be composed of an image sensor (such as an RGB sensor or an IR sensor) that captures an image of the event detection area 250. For example, when a detection value becomes equal to or greater than a predetermined threshold, the sensor unit 273 detects the occurrence of an event and notifies the control unit 272.

[0167] In addition, the sensor unit 273 may be arranged in close proximity to the transmission unit 271, or may be arranged separately, for example, with the transmission unit 271 being arranged at a high location closest to the satellite 21 and the sensor unit 273 being arranged at a low location closer to the ground.

[0168] A plurality of sensors of different types or a plurality of sensors of the same type may be mounted on one transmitting device 251. When a plurality of sensors are mounted on the transmitting device 251, it may be necessary to transmit the sensor detection result by adding sensor information such as the sensor detection range as the imaging target position and the sensor detection type as the transmission information.

[0169] The power supply unit 274 is formed of, for example, a battery that is charged by solar power generation or the like, and supplies power to each unit of the transmission device 251 .

[0170] The transmitting device 251 is configured as described above and is a communication device that enables only one-way communication from the transmitting device 251 to the satellite 21, but it may also be a communication device that enables two-way communication including from the satellite 21 to the transmitting device 251.

[0171] In both one-way and two-way communication, in the case of omnidirectionality, the transmitting side does not need to orient the antenna toward the receiving side, satellite 21 or earth station 15, and is therefore particularly suitable for transmitting from the ground to satellite 21 in the sky. In this embodiment, the transmitting unit 271 of transmitting device 251 is omnidirectional, and transmitting device 251 is a device that performs one-way communication, but of course transmitting device 251 may also be a device that has directionality and performs two-way communication.

[0172] 10. First Event Imaging Sequence of the Second Embodiment Next, the first event imaging sequence performed by the satellite image processing system 1 of the second embodiment will be described with reference to the flowchart of FIG.

[0173] First, in step S141, the control unit 272 of the transmission device 251 determines whether an event has been detected by the sensor unit 273. When the sensor unit 273 detects a predetermined event and notifies the control unit 272 of the occurrence of the event, the control unit 272 determines that the event has been detected. Therefore, in step S141, the control unit 272 waits until it receives a notification of the occurrence of an event from the sensor unit 273, and if it is determined that the event has been detected, the process proceeds from step S141 to step S142.

[0174] In response to the occurrence of the event, in step S142, the control unit 272 controls the transmission unit 271 to transmit an image capturing instruction to a satellite 21 passing near the transmission device 251. The transmission unit 271 transmits the image capturing instruction in response to a command from the control unit 272.

[0175] Since the communication between the transmitting device 251 and the satellite 21 is one-way communication only from the ground to the satellite 21, the transmitting device 251 cannot confirm whether the satellite 21 has received the image capturing instruction. Therefore, the transmitting device 251 continues to transmit the image capturing instruction for a fixed period of time, such as 30 minutes or 1 hour, or transmits the image capturing instruction intermittently at fixed time intervals. If the transmitting device 251 and the satellite 21 are capable of two-way communication, the transmitting device 251 may receive a reception completion from the satellite 21 and stop transmitting the image capturing instruction, as in the image capturing sequence described with reference to Fig. 6. The reception completion from the satellite 21 to the transmitting device 251 may include a message indicating that the satellite 21 will capture an image.

[0176] Furthermore, in this imaging sequence, when the transmitting device 251 detects the occurrence of an event, it transmits an imaging instruction without selecting a satellite 21. However, if the orbital information and imaging capability of the satellite 21 passing overhead are known, the imaging instruction may be transmitted by specifying the satellite group 31 or satellite 21 that satisfies the required imaging conditions by group ID or individual ID.

[0177] Imaging-related information such as requested imaging conditions, requested imaging target position, sensor ID, event occurrence time, and detected event type is added as parameters to the imaging instruction from the transmitting device 251 to the satellite 21 and transmitted. The requested imaging conditions include, for example, resolution and wavelength band (RGB, IR, etc.). The requested imaging target position represents an area on the ground of the imaging target, and is a position corresponding to an event occurrence area of ​​the sensor unit 273. The installation position of the transmitting device 251 or the sensor unit 273 may be stored as the requested imaging target position. The sensor ID is sensor identification information for identifying the sensor unit 273 that detected the event. The event occurrence time is the time when the sensor unit 273 detected the event, and corresponds to the time when a request for imaging instruction was generated. The detected event type represents the type of event detected by the sensor unit 273, such as detection of abnormal temperature. The detected event type may store the type of sensor instead of a specific type of detected event.

[0178] In step S161, the satellite 21 receives an imaging instruction from the transmission device 251, and in step S162, determines whether imaging is possible by itself. The satellite 21 checks whether the requested imaging conditions added to the imaging instruction are satisfied, and determines whether imaging is possible by itself. If it is determined in step S162 that imaging is not possible by itself, the satellite 21 ends the process.

[0179] On the other hand, if it is determined in step S162 that the satellite 21 is capable of capturing an image by itself, the process proceeds to step S163, and the satellite 21 performs an image capturing preparation process based on the received image capturing instruction. Next, in step S164, the satellite 21 captures an image, and in step S165, generates metadata and adds it to the captured image. The processes in steps S163 to S165 are basically the same as the processes in steps S33 to S35 in FIG. 6 described above, and therefore details will be omitted. The metadata may include a part or all of the information received from the transmission device 251. For example, the metadata may include information such as a sensor ID representing the sensor unit 273 and the time of event occurrence.

[0180] In step S166, the satellite 21 determines whether it has reached a downlink point, in other words, whether it has reached a range in which it can communicate with the communication device 13 of the ground station 15. The satellite 21 repeats the process of step S166 until it is determined that it has reached the downlink point, and if it is determined that it has reached the downlink point, the process proceeds to step S167.

[0181] In step S167, the satellite 21 transmits (downlinks) the captured image with the metadata added to the ground station 15. The downlink may be performed via the relay satellite 22.

[0182] In step S181, the management system receives captured images from the satellites 21. That is, the communication device 13 receives the captured images via the antenna 14 and supplies them to the satellite group management device 11. After receiving the captured images, the management system performs the same processes as steps S17 to S19 in Fig. 6, but a redundant description will be omitted.

[0183] <11. Second Event Imaging Sequence of the Second Embodiment> Next, a second event imaging sequence performed by the satellite image processing system 1 of the second embodiment will be described with reference to the flowchart of FIG.

[0184] In the first event imaging sequence described above, each satellite 21 individually determined whether imaging was possible, and if imaging was performed, transmitted the captured image to the communication device 13 on the ground.

[0185] 14, a process is added in which, when a satellite 21 that has received an imaging instruction determines that it cannot perform imaging by itself, a subsequent satellite 21 takes over the imaging instruction. The subsequent satellite 21 is, for example, a satellite 21 that belongs to the same satellite group 31 that is operated in a constellation or formation flight. In the second event imaging sequence below, the satellite 21 that receives the imaging instruction is distinguished as the first satellite 21, and the subsequent satellite 21 that takes over the imaging instruction is distinguished as the second satellite 21.

[0186] The detection of the event occurrence in steps S141 and S142 by the transmission device 251 and the transmission of the imaging instruction are the same as those in the first event imaging sequence described above.

[0187] In step S201, the first satellite 21 receives an imaging instruction from the transmission device 251, and in step S202, determines whether imaging by itself is possible. If it is determined in step S202 that imaging by itself is possible, the process proceeds to step S203, where the first satellite 21 performs imaging and transmission based on the imaging instruction, and ends the process. The imaging sequence when it is determined that imaging by itself is possible is the same as the first event imaging sequence described above, and therefore description thereof will be omitted.

[0188] On the other hand, if it is determined in step S202 that imaging by the first satellite 21 is not possible, the process proceeds to step S204, where the first satellite 21 determines whether imaging is possible by a subsequent second satellite 21 that belongs to the first satellite constellation 31. If it is determined in step S204 that imaging by the second satellite 21 is not possible, the process ends.

[0189] If it is determined in step S204 that imaging by the second satellite 21 is possible, the process proceeds to step S205, in which the first satellite 21 transmits an imaging instruction to the subsequent second satellite 21 through inter-satellite communication.

[0190] Then, in step S206, the first satellite 21 determines whether or not the downlink point has been reached, and repeats the process of step S206 until it is determined that the downlink point has been reached.

[0191] Then, if it is determined in step S206 that the downlink point has been reached, the process proceeds to step S207, where the first satellite 21 transmits (downlinks) to the ground station 15 the event detection data included in the imaging instruction received from the transmission device 251. The event detection data includes some or all of the imaging related information included in the imaging instruction, information indicating that the imaging instruction has been transferred to the subsequent satellite, and information indicating the subsequent second satellite 21 to which the imaging instruction has been transferred. As with the other processes described above, the downlink may be performed via the relay satellite 22. The process of the first satellite 21 then ends.

[0192] The second satellite 21, which is the succeeding satellite to which the imaging instruction has been transmitted from the first satellite 21 by inter-satellite communication, receives the imaging instruction in step S221, and performs imaging preparation processing based on the received imaging instruction in step S222.

[0193] The processes of steps S223 to S226 are similar to the processes of steps S164 to S167 in Fig. 13. Through the processes of steps S223 to S226, imaging is performed and the captured image and metadata are generated. When the captured image reaches the downlink point, the captured image with the metadata is transmitted to the ground station 15.

[0194] Meanwhile, the management system receives the event detection data in step S241 in response to the transmission of the event detection data by the first satellite 21. Also, in step S242, the management system receives the captured image in response to the transmission of the captured image by the second satellite 21. After receiving the captured image, the management system performs the same processes as steps S17 to S19 in Fig. 6, but a description thereof will be omitted to avoid duplication.

[0195] <12. Third Event Imaging Sequence of the Second Embodiment> Next, a third event imaging sequence performed by the satellite image processing system 1 of the second embodiment will be described with reference to the flowchart of FIG.

[0196] In the second event imaging sequence described above, an imaging instruction was transferred from the first satellite 21 to the second satellite 21 using inter-satellite communication, whereas the third event imaging sequence is an example in which an imaging instruction is transferred from the first satellite 21 to the second satellite 21 by communication via the ground station 15.

[0197] The detection of the event occurrence in steps S141 and S142 by the transmission device 251 and the transmission of the imaging instruction are the same as those in the first event imaging sequence described above.

[0198] In step S301, the first satellite 21 receives an imaging instruction from the transmission device 251, and in step S302, determines whether imaging by itself is possible. If it is determined in step S302 that imaging by itself is possible, the process proceeds to step S303, where the first satellite 21 performs imaging and transmission based on the imaging instruction, and ends the process. The imaging sequence when it is determined that imaging by itself is possible is the same as the first event imaging sequence described above, and therefore description thereof will be omitted.

[0199] On the other hand, if it is determined in step S302 that imaging by the first satellite 21 is not possible, the process proceeds to step S304, where the first satellite 21 determines whether imaging by a subsequent satellite 21 belonging to its own satellite group 31 is possible. If it is determined in step S304 that imaging by the subsequent satellite 21 is possible, the process proceeds to step S305, where imaging and transmission by the subsequent satellite 21 is performed, and the process ends. The imaging sequence when it is determined that imaging by the subsequent satellite 21 is possible is the same as the second event imaging sequence described above, and therefore description thereof will be omitted.

[0200] If it is determined in step S304 that imaging by the subsequent satellite 21 is not possible, processing proceeds to step S306, in which the first satellite 21 determines whether it has reached the downlink point, and repeats the processing of step S306 until it is determined that it has reached the downlink point.

[0201] Then, if it is determined in step S306 that the downlink point has been reached, the process proceeds to step S307, where the first satellite 21 transmits (downlinks) the imaging instruction received from the transmitting device 251 to the ground station 15. As with the other processes described above, the downlink may be performed via the relay satellite 22. The process of the first satellite 21 then ends.

[0202] In step S321, the management system receives the imaging instruction in response to the transmission of the imaging instruction by the first satellite 21. Then, in step S322, the management system identifies another satellite 21 that satisfies the imaging request based on the requested imaging conditions, requested imaging target position, etc. included in the imaging related information of the imaging instruction. Here, the second satellite 21 is identified as the other satellite 21.

[0203] In step S323, the management system transmits an imaging instruction to the identified second satellite 21. Note that the ground station 15 (the communication device 13) that receives the imaging instruction from the first satellite 21 and the ground station 15 (the communication device 13) that transmits the imaging instruction to the second satellite 21 may be the same or different.

[0204] In step S341, the second satellite 21 receives an image capture instruction from the ground station 15. The subsequent processes in steps S342 to S346 are similar to the processes in steps S222 to S226 in Fig. 14, and therefore will not be described. In step S346, the captured image is transmitted from the second satellite 21 to the management system.

[0205] In step S324, the management system receives the captured image, and the third event imaging sequence ends.

[0206] In the third event imaging sequence described above, the first satellite 21 transmits an imaging instruction to the ground station 15 when it is determined that imaging by the subsequent satellite 21 is not possible. However, the first satellite 21 may transmit an imaging instruction to the ground station 15 when it is determined that imaging by itself is not possible, without determining whether imaging by the subsequent satellite 21 is possible.

[0207] According to the third event imaging sequence, even if the requested imaging target location is a place that cannot be connected to a network, such as on the ocean, imaging instructions can be transmitted to the management system via the first satellite 21, and imaging can be performed by the second satellite 21.

[0208] <13. Other configuration examples of the transmitting device> 12 is integrated with a transmission unit that transmits an image capture instruction and has a built-in sensor that detects the occurrence of an event. However, the sensor that detects the occurrence of an event and the transmission device that transmits an image capture instruction can be configured as separate devices.

[0209] FIG. 16 is a block diagram showing another example of the configuration of the transmission device in the second embodiment. In FIG.

[0210] In the event detection area 250 (FIG. 11), a transmitting device 291, a control device 292, and one or more sensors 293 are installed. Although FIG. 16 shows an example in which the number of sensors 293 is three, namely, sensors 293A to 293C, the number of sensors 293 is arbitrary. Furthermore, a plurality of sets of a transmitting device 291, a control device 292, and one or more sensors 293 may be installed in the event detection area 250.

[0211] The transmitting device 291 , under the control of the control device 292 , transmits an image capturing instruction to a satellite 21 passing near the transmitting device 291 .

[0212] When a predetermined event is detected by any of the multiple sensors 293 (293A to 293C), the control device 292 acquires the event detection result from the sensor 293, generates an imaging instruction, and controls the transmission device 291 to transmit the imaging instruction. As in the above example, imaging related information is added as a parameter to this imaging instruction.

[0213] Each of the multiple sensors 293 (293A to 293C) corresponds to the above-mentioned sensor unit 273, detects the occurrence of an event, and notifies the control device 292. The multiple sensors 293 may be composed of sensors of different types, or may be sensors of the same type. The multiple sensors 293 may be arranged close to each other, or may be arranged at a distance. Furthermore, the multiple sensors 293 may be arranged close to the transmission device 291 and the control device 292, or may be arranged at a distance. The above-mentioned sensor information is added to the notification of the occurrence of an event from the sensor 293 to the control device 292 as necessary.

[0214] In the satellite image processing system 1 of the second embodiment, even if the transmitting device 291 and the sensor 293 are configured as separate devices as described above, the first to third event imaging sequences described above can be executed in the same manner.

[0215] <14. Application example of satellite image processing system using event detection sensor> An application example of a satellite image processing system using the event detection sensor according to the second embodiment will be described below.

[0216] (1) Event detection in farmland A plurality of sensors (transmitting device 251 including sensor unit 273 or sensor 293) are installed at regular intervals in a predetermined observation area of ​​the farmland, and each of the plurality of sensors detects an abnormality such as the occurrence of pests or diseases. The transmitting device 251 or 291 transmits an imaging instruction to the satellite 21 according to the detection result of the abnormality in the farmland as an event. The satellite 21 performs, for example, RGB imaging, or R (Red) and IR (Infrared) imaging for vegetation index such as NDVI. The sensor detection range of the sensor that detected the abnormality is assigned to the requested imaging target position added to the imaging instruction. The satellite 21 that receives the imaging instruction may image only the sensor detection range of the sensor in which the abnormality occurred among the observation area in which the plurality of sensors are arranged, or may image the entire observation area in a wide area. In addition, the imaging conditions such as zoom may be changed to perform both imaging of the sensor detection range of the sensor that detected the abnormality and wide area imaging of the entire observation area. Instead of detecting an abnormality, a command to satellite 21 to take a photograph can also be issued when a specified condition occurs to check the growth status, such as the earth's surface reaching a specified environmental state (for example, the earth's surface temperature reaching a specified temperature), the amount of photosynthesis or growth status of a plant reaching a specified state, or germination is detected.

[0217] (2) Event detection in the ocean For example, a buoy incorporating a transmitting device 251 including a sensor unit 273 is released into a target ocean area. The sensor unit 273 detects predetermined conditions such as fish detection, seawater temperature, ocean current speed, or wind speed, and the transmitting device 251 transmits an image capture instruction to the satellite 21 based on the event detection result. The image capture related information in the image capture instruction includes the requested image capture conditions, the requested image capture target position, the event occurrence time, and the like. Since there are only a limited number of satellites 21 that can capture images of nighttime conditions, a satellite 21 is selected based on the requested image capture conditions, and the situation of the target ocean area is analyzed based on the captured images.

[0218] (3) Observation of uninhabited areas A sensor (transmitting device 251 including a sensor unit 273, or a sensor 293) is installed in an uninhabited area such as a forest, a mountain, or a desert, and detects anomalies such as changes in weather conditions, detection of organisms to be observed, and forest fires. The satellite 21 captures images based on an image capture command from the transmitting device 251 or 291. The situation in the uninhabited area is analyzed based on the captured images.

[0219] (4) Accident monitoring For example, the transmitter 251 is mounted on the black box of an airplane or on a ship, and transmits an image capturing command in the event of an emergency such as an airplane crash, a ship running aground, or an oil tanker leak. The satellite 21 quickly captures an image of the location where the emergency occurred and transmits the image to the ground station 15.

[0220] (5) Climber in distress A mountain climber or the like carries a transmission device 251, and in the event of a disaster, an imaging instruction including a distress signal as a detected event type and imaging related information including the location of the disaster as a requested imaging target position is transmitted from the transmission device 251 to the satellite 21. The satellite 21 takes an image of the location of the disaster based on the imaging instruction and transmits it to the ground station 15.

[0221] (6) Pipeline Emission Control Sensors are attached to the pipeline at predetermined intervals to monitor the occurrence of leakage. When a leakage is detected, an imaging instruction is sent to the satellite 21. The imaging instruction is sent with imaging-related information specifying a satellite 21 capable of detecting leakage, such as a satellite 21 capable of heat detection in the IR band, as a required imaging condition, and the satellite 21 that satisfies the request performs imaging. Based on the captured images, the leakage situation in the leakage area can be quickly observed. In particular, when the leakage from the pipeline is caused by human factors, prompt observation after the occurrence of the event is effective.

[0222] (7)Other The captured image triggered by the sensor 293 arranged on the ground may be treated as primary information, and the captured image may be combined with other images for image analysis, etc. For example, the captured image triggered by the sensor 293 is captured promptly by a low-performance satellite 21, prioritizing the timing of capture. Thereafter, the satellite group management device 11 schedules the satellite 21 with higher imaging capability to capture high-resolution, high-precision images. The satellite group management device 11 performs analysis using a first captured image captured by the low-performance satellite 21 and a second captured image captured by the satellite 21 with higher imaging capability. The satellite group management device 11 may, for example, increase the resolution of the first captured image based on the difference information, or perform a synthesis process of the first captured image and the second captured image.

[0223] As described above, satellite remote sensing using a sensor allows an event occurring on the ground to be detected by the sensor, and an image capturing command can be sent directly to the satellite 21 in the sky. In particular, even a sensor installed in an area not connected to the Internet, such as the ocean, can directly send an image capturing command to a satellite, or send an image capturing command to another satellite via the satellite. For example, an event occurring at a specific location within a vast area can be instantly detected and an image captured, greatly reducing the amount of work required.

[0224] <15. Computer configuration example> The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed in a computer. Here, the computer includes a microcomputer built into dedicated hardware, and a general-purpose personal computer, for example, capable of executing various functions by installing various programs.

[0225] FIG. 17 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes by a program.

[0226] In the computer, a central processing unit (CPU) 301, a read only memory (ROM) 302, and a random access memory (RAM) 303 are interconnected via a bus 304.

[0227] An input / output interface 305 is further connected to the bus 304. An input unit 306, an output unit 307, a storage unit 308, a communication unit 309, and a drive 310 are connected to the input / output interface 305.

[0228] The input unit 306 includes a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 307 includes a display, a speaker, an output terminal, etc. The storage unit 308 includes a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 309 includes a network interface, etc. The drive 310 drives a removable recording medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0229] In the computer configured as above, the series of processes described above are performed by the CPU 301, for example, loading a program stored in the storage unit 308 into the RAM 303 via the input / output interface 305 and the bus 304 and executing the program. The RAM 303 also stores data and the like necessary for the CPU 301 to execute various processes as appropriate.

[0230] The program executed by the computer (CPU 301) can be provided by being recorded on a removable recording medium 311 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0231] In the computer, the program can be installed in the storage unit 308 via the input / output interface 305 by mounting the removable recording medium 311 in the drive 310. The program can also be received by the communication unit 309 via a wired or wireless transmission medium and installed in the storage unit 308. Alternatively, the program can be installed in the ROM 302 or the storage unit 308 in advance.

[0232] In this specification, the steps described in the flowcharts may be performed chronologically in the order described, but they do not necessarily have to be processed chronologically and may be performed in parallel or at the required timing, such as when a call is made.

[0233] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same case. Therefore, multiple devices housed in separate cases and connected via a network, and a single device in which multiple modules are housed in a single case, are both systems.

[0234] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0235] For example, it is possible to adopt a configuration in which all or part of the above-described multiple embodiments are combined.

[0236] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0237] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0238] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0239] It should be noted that the effects described in this specification are merely examples and are not limiting, and there may be effects other than those described in this specification.

[0240] The present technology can have the following configurations. (1) A management device for managing images captured by a satellite, Metadata including at least information about a person involved in the captured image is added to the captured image. Image management methods. (2) The related party information includes information on the owner of the satellite, the operator of a service using the satellite, or the right holder of the captured image. The image management method according to (1) above. (3) The management device applies a process of limiting use of the captured image. The image management method according to (1) or (2) above. (4) The management device performs watermark processing on the captured image. The image management method according to (1) or (2) above. (5) The management device receives the captured image with the metadata added thereto from the satellite. The image management method according to any one of (1) to (4). (6) The management device receives the metadata from the satellite as a stream separate from the captured images. The image management method according to any one of (1) to (4). (7) The metadata includes a satellite identifier that identifies the satellite, and a satellite constellation identifier that identifies a constellation that includes the satellite. The image management method according to any one of (1) to (6). (8) The metadata includes relative position information for each satellite constellation. The image management method according to any one of (1) to (7) above. (9) The relative position information includes the order of arrangement of the satellites constituting the satellite group and the distances between the satellites. The image management method according to (8) above. (10) The metadata includes information regarding the image content of the satellite imagery. The image management method according to any one of (1) to (9). (11) The information on the imaging contents includes at least one of imaging target position information, imaging conditions, image sensor type, imaging time, and satellite position at the time of imaging. The image management method according to (10) above. (12) The metadata includes information regarding the image type of the captured image. The image management method according to any one of (1) to (11) above. (13) The information regarding the image type of the captured image includes image processing information. The image management method according to (12) above. (14) A data structure of metadata for an image captured by a satellite, The metadata includes at least information about a person involved in the captured image, A management device for managing the captured images, Used in the process of matching the people involved in the captured image Metadata data structure. (15) The metadata is attached to the captured image. The data structure of the metadata described in (14) above. [Explanation of symbols]

[0241] 1 satellite image processing system, 11 satellite group management device, 13 communication device, 14 antenna, 15 ground station (base station), 21 satellite, 31 satellite group, 41 information providing server, 42 image analysis server, 101 management unit, 111 imaging device, 211 control unit, 222 control unit, 231 control unit, 250 event detection area, 251 transmission device, 271 transmission unit, 272 control unit, 273 sensor unit, 291 transmission device, 292 control device, 293 sensor, 301 CPU, 302 ROM, 303 RAM, 306 input unit, 307 output unit, 308 memory unit, 309 communication unit, 310 drive

Claims

1. A management device that manages images captured by satellites operated by formation flight, Metadata including at least a satellite group identifier for identifying a satellite group of a formation flight including the satellite, a satellite identifier for identifying the satellite, and information on a person involved in the captured image is added to the captured image. Image management methods.

2. The related party information includes information on the owner of the satellite, the operator of a service using the satellite, or the right holder of the captured image. The image management method according to claim 1 .

3. The management device applies a process of limiting use of the captured image. The image management method according to claim 1 .

4. The management device performs watermark processing on the captured image. The image management method according to claim 1 .

5. The management device receives the captured image with the metadata added thereto from the satellite. The image management method according to claim 1 .

6. The management device receives the metadata from the satellite as a stream separate from the captured images. The image management method according to claim 1 .

7. The metadata includes relative position information for each satellite constellation. The image management method according to claim 1 .

8. The relative position information includes the order of arrangement of the satellites constituting the satellite group and the distances between the satellites. The image management method according to claim 7.

9. The metadata includes information regarding the image content of the satellite imagery. The image management method according to claim 1 .

10. The information on the imaging contents includes at least one of imaging target position information, imaging conditions, image sensor type, imaging time, and satellite position at the time of imaging. The image management method according to claim 9.

11. The metadata includes information regarding the image type of the captured image. The image management method according to claim 1 .

12. The information regarding the image type of the captured image includes image processing information. The image management method according to claim 11.

Images