Display control devices, display control methods and program products
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-01-06
- Publication Date
- 2026-08-14
Smart Images
Figure CN114946176B_ABST
Abstract
Description
Technical Field
[0001] This technology relates to display control devices, display control methods and programs, and more specifically to display control devices, display control methods and programs that enable real-time checking of images. Background Technology
[0002] Remote sensing is performed by using observation satellites equipped with imaging devices to image the ground and observe the condition of target areas or objects (see, for example, PTL 1 and PTL 2). In particular, the number of small observation satellites operating in low Earth orbit has been increasing in recent years.
[0003] Reference List
[0004] Patent documents
[0005] PTL 1: JP 2000-111359 A
[0006] PTL 2: JP 2006-115283 A Summary of the Invention
[0007] Technical issues
[0008] When a satellite passes over a ground station, the satellite images captured by the observation satellite are transmitted downlink. If the image data transmission capacity is large, communication delays will occur, making it difficult to verify the images in real time.
[0009] This technology was proposed in view of such circumstances and enables real-time image verification.
[0010] Solution to the problem
[0011] According to one aspect of the present technology, a display control device is provided, comprising: a receiving unit that receives small-capacity data as information about current imaging of an artificial satellite; and a control unit that displays a real-time view image based on the small-capacity data on the display unit, wherein the receiving unit receives satellite images corresponding to the real-time view image as large-capacity data at a timing different from that of the small-capacity data.
[0012] According to one aspect of the present technology, a display control method is provided to allow a display control device to perform: receiving small-capacity data as information about the current imaging of an artificial satellite; displaying a real-time view image based on the small-capacity data on a display unit; and receiving a satellite image corresponding to the real-time view image as large-capacity data at a timing different from that of the small-capacity data.
[0013] According to one aspect of the present technology, a program is provided for causing a computer to perform: receiving small-volume data as information about current imaging of an artificial satellite; displaying a real-time view image based on the small-volume data on a display unit; and receiving satellite images corresponding to the real-time view image as large-volume data at a timing different from that of the small-volume data.
[0014] In one aspect of this technology, a small amount of data is received as information about the current imaging of an artificial satellite, a real-time view image based on the small amount of data is displayed on a display unit, and a large amount of data is received as satellite images corresponding to the real-time view image at a timing different from that of the small amount of data.
[0015] According to one aspect of this technology, the display control device can be implemented by causing a computer to execute a program. The program to be executed by the computer can be provided by transmitting via a transmission medium or by recording on a recording medium.
[0016] The display control device can be a standalone device or an internal block that forms part of a device. Attached Figure Description
[0017] Figure 1 This is a block diagram illustrating a configuration example of a satellite image processing system according to an embodiment of the present technology.
[0018] Figure 2 It is a block diagram showing the functional configuration of satellite management equipment and artificial satellites.
[0019] Figure 3 This is the flowchart for the first real-time viewfinder image display process.
[0020] Figure 4 This is a diagram illustrating the processing based on the user's control instructions.
[0021] Figure 5 This is a flowchart of the second real-time viewfinder image display process.
[0022] Figure 6 This is a block diagram illustrating a configuration example of a computer to which the present technology is applied. Detailed Implementation
[0023] The embodiments of this technology will now be described with reference to the accompanying drawings. Furthermore, components with substantially the same functional configuration will be indicated by the same reference numerals in this specification and the drawings, and therefore repeated descriptions thereof will be omitted. The descriptions will proceed in the following order.
[0024] 1. Configuration example of a satellite image processing system
[0025] 2. First Real-Time View Image Display Processing
[0026] 3. Second Real-Time View Image Display Processing
[0027] 4. Application Examples
[0028] 5. Computer Configuration Example
[0029] <1. Configuration Example of Satellite Image Processing System>
[0030] Figure 1 This is a block diagram illustrating a configuration example of a satellite image processing system according to an embodiment of the present technology.
[0031] Figure 1 The satellite image processing system 1 is a system that enables ground-based equipment to review satellite images captured by one or more artificial satellites 21 (hereinafter referred to as satellite 21) in real time. In this embodiment, satellite 21 is equipped with imaging equipment and has at least the function of imaging the ground.
[0032] The satellite operating company has satellite management equipment 11 for managing multiple satellites 21 and multiple communication devices 13 for communicating with the satellites 21. Some of the satellite management equipment 11 and the multiple communication devices 13 may be owned by a company other than the satellite operating company. The satellite management equipment 11 and the multiple communication devices 13 are interconnected via a predetermined network 12. The communication devices 13 are located in a ground station (ground base station) 15. Note that... Figure 1 An example is shown where the number of communication devices 13 is three (i.e., communication devices 13A to 13C), but the number of communication devices 13 is arbitrary.
[0033] Satellite management device 11 manages multiple satellites 21 owned by a satellite operating company. Specifically, satellite management device 11 obtains relevant information from information providing servers 41 of one or more external organizations as needed, and determines the operation plan for the multiple satellites 21 owned by satellite management device 11. Then, in response to a customer's request, satellite management device 11 causes the designated satellite 21 to perform imaging by sending an imaging start instruction to the designated satellite 21 via communication device 13. Satellite management device 11 acquires, displays, or stores satellite images transmitted from satellites 21 via communication device 13. After performing predetermined image processing on the acquired satellite images as needed, the acquired satellite images can be provided (transmitted) to the customer. The acquired satellite images can be provided (transmitted) to the image analysis server 42 of an image analysis company, and can be provided to the customer after performing predetermined image processing on the acquired satellite images.
[0034] An information providing server 41, located within an external organization, responds to requests from satellite management equipment 11 or periodically provides predetermined relevant information to satellite management equipment 11 via a predetermined network. The relevant information provided by the information providing server 41 includes, for example, satellite orbit information described in Two Line Elements (TLE) format (hereinafter referred to as TLE information) as relevant information can be obtained from the North American Aerospace Defense Command (NORAD), an external organization. Similarly, meteorological information such as weather and cloud cover at predetermined locations on Earth can be obtained from a meteorological information providing company, also an external organization.
[0035] Image analysis server 42 performs predetermined image processing on satellite images captured by satellite 21, provided via a predetermined network from satellite management equipment 11. The processed images are then provided to either the image analysis company's clients or to the satellite management equipment 11 of the satellite operating company. For example, image analysis server 42 performs metadata generation processing to add predetermined metadata to the satellite images captured by satellite 21, correction processing such as distortion correction of the satellite images, and image compositing processing such as color compositing. Image processing of the satellite images can be performed by the satellite operating company, and in this case, the satellite operating company and the image analysis company are the same entity. Satellite management equipment 11 and image analysis server 42 can be implemented using a single device.
[0036] Under the control of satellite management equipment 11, communication equipment 13 communicates with a predetermined satellite 21 designated by satellite management equipment 11 via antenna 14. For example, communication equipment 13 sends an imaging start instruction to predetermined satellite 21 to begin imaging a predetermined area on the ground. Communication equipment 13 receives satellite images transmitted from satellite 21 and provides these satellite images to satellite management equipment 11 via network 12. The transmission from communication equipment 13 at ground station 15 to satellite 21 is also referred to as the uplink, and the transmission from satellite 21 to communication equipment 13 is also referred to as the downlink. Communication equipment 13 can communicate directly with satellite 21 and can also communicate with relay satellite 22. For example, a geostationary satellite is used as relay satellite 22.
[0037] Network 12, or the network between information providing server 41 or image analysis server 42 and satellite management device 11, is any communication network. This communication network can be a wired communication network or a wireless communication network, or a combination of both. Network 12 and the network between information providing server 41 or image analysis server 42 and satellite management device 11 can be configured by one communication network or by multiple communication networks. These networks can be communication networks or communication channels of any communication standard, such as, for example, the Internet, public telephone networks, wide area communication networks for wireless mobile vehicles (such as so-called 4G and 5G lines), wireless communication networks performing communication conforming to WAN (Wide Area Network), LAN (Local Area Network), and Bluetooth (trademark) standards, communication channels for short-range wireless communication such as NFC (Near Field Communication), communication channels for infrared communication, and communication networks for wired communication conforming to standards such as HDMI (High Definition Multimedia Interface) and USB (Universal Serial Bus).
[0038] Each satellite 21 can operate individually or in groups. Multiple satellites 21 operating in groups constitute a satellite group 31. Figure 1 In this constellation, satellites 21A and 21B operate individually, while satellites 21C and 21D form a satellite group 31A. Figure 1 In the example, for simplicity, an example of a satellite group 31 consisting of two satellites 21 is shown, but the number of satellites 21 constituting a satellite group 31 is not limited to two.
[0039] Systems that operate multiple satellites 21 as a unit (satellite group 31) include constellations and formation flying. A constellation is a system that deploys services primarily globally by launching a large number of satellites 21 into one or more orbital planes. Even individual satellites have predetermined functions, and multiple satellites 21 are operated for purposes such as increasing observation frequency. On the other hand, formation flying is a system in which multiple satellites 21 are deployed over a narrow area of approximately several kilometers while maintaining relative positional relationships. Formation flying can provide services that are not possible with a single satellite, such as high-precision 3D (three-dimensional) measurement and velocity detection of moving objects. In this embodiment, whether the operation of the satellite group is constellation or formation flying is irrelevant.
[0040] When communication device 13 communicates with each satellite 21, it can use methods such as direct communication with satellite 21 as in satellites 21A and 21B, and indirect communication with satellite 21C (which is another satellite 21) through inter-satellite communication as in satellite 21D. Indirect communication methods include communication via relay satellite 22. The method for communicating with ground station 15 (communication device 13 of ground station 15) can be predetermined by satellite 21, or appropriately selected based on the content of the communication.
[0041] In the satellite image processing system 1 configured as described above, satellite 21, acting as an observation satellite, begins imaging a predetermined location on the ground based on imaging data from satellite management equipment 11. If the data of the image captured by satellite 21 is transmitted as is, a delay occurs due to the large data capacity, and it is difficult to observe the image in real time as a live view image.
[0042] Therefore, the satellite image processing system 1 is configured to enable the ground-based satellite management equipment 11 to observe images similar to those captured by the satellite 21 in real time.
[0043] Figure 2 This is a block diagram showing the functional configuration of the satellite management device 11 and the satellite 21 for observing images captured by the satellite 21.
[0044] Satellite 21 includes an antenna AT, a satellite communication unit 61, an imaging device 62, a control unit 63, and a storage unit 64. Satellite 21 is primarily configured for image-related functions only, and although not shown, it is also equipped with propulsion devices for attitude control (e.g., solid-state motors and ion engines), sensors for position control (e.g., a GPS receiver, a star tracker (attitude sensor), an accelerometer, and a gyroscope sensor), and power sources (e.g., batteries and solar panels).
[0045] Based on the control unit 63, the satellite communication unit 61 transmits image data of the image captured by the imaging device 62, status data indicating the status of the satellite 21 during imaging, etc., to the communication device 13 of the ground station 15 via the antenna AT. The data transmitted from the satellite communication unit 61 to the communication device 13 is then provided from the communication device 13 to the satellite management device 11.
[0046] Imaging device 62 may consist of, for example, a camera module including an image sensor, and image the object based on the control of control unit 63. When satellite 21 is a synthetic aperture radar (SAR) satellite, imaging device 62 may consist of radar equipment.
[0047] Control unit 63 controls the operation of the entire satellite 21. For example, based on an imaging start instruction from satellite management device 11, control unit 63 causes imaging device 62 to perform imaging. Control unit 63 stores the satellite images obtained through imaging as large-capacity data in storage unit 64, and also performs capacity reduction processing to generate smaller-capacity data with a smaller capacity than the satellite images. For example, the smaller-capacity data includes images obtained by converting the satellite images captured by imaging device 62 into smaller-capacity data, and status data indicating the state of satellite 21 when imaging device 62 performs imaging.
[0048] Storage unit 64 stores the control program and parameters executed by control unit 63. Storage unit 64 stores image data (large-capacity data) captured by imaging device 62 and small-capacity data generated by control unit 63, and provides it to satellite communication unit 61 or control unit 63 as needed.
[0049] The satellite management device 11 includes a control unit 81, a communication unit 82, a storage unit 83, an operation unit 84, and a display unit 85.
[0050] Control unit 81 manages multiple satellites 21 owned by the satellite operating company by executing a satellite management application stored in storage unit 83. For example, control unit 81 uses relevant information obtained from information providing server 41 as needed to determine the operation plan for multiple satellites 21, and sends attitude control signals and imaging instructions to each satellite 21 via communication device 13. Based on large-volume and small-volume data of satellite images transmitted from satellites 21 via communication device 13, control unit 81 performs processes such as displaying observed images on display unit 85.
[0051] According to the instructions of the control unit 81, the communication unit 82 performs predetermined communication with the communication device 13 via the network 12 and performs predetermined communication with the image analysis server 42. For example, the communication unit 82 receives large-capacity data and small-capacity data related to satellite images transmitted from the satellite 21.
[0052] Storage unit 83 stores large-capacity data and small-capacity data related to satellite images transmitted from satellite 21, according to the instructions of control unit 81.
[0053] The operation unit 84 may consist of, for example, a keyboard, mouse, touch panel, etc., and receives commands and data inputs based on user (operator) operations and provides them to the control unit 81.
[0054] The display unit 85 is, for example, composed of an LCD (liquid crystal display) or an organic EL (electroluminescent) display. The display unit 85 displays a screen for a satellite management application, showing satellite images based on large amounts of data transmitted from satellite 21, real-time view images based on small amounts of data transmitted from satellite 21, etc.
[0055] The satellite management device 11 serves as a display control device, which displays satellite images captured by the imaging device 62 of the satellite 21 on a predetermined display unit (display unit 85 or an external display device) based on user operations.
[0056] <2. First Real-Time View Image Display Processing>
[0057] Next, we will refer to Figure 3 The flowchart describes the first real-time view image display process for displaying real-time view images used for real-time observation. For example, this process begins when a user operating the satellite management device 11 performs a real-time view image display start operation.
[0058] First, in step S11, the control unit 81 of the satellite management device 11 sends an imaging start instruction to the satellite 21 via the communication unit 82.
[0059] In step S41, the control unit 63 of satellite 21 receives an imaging start instruction sent from satellite management device 11 via satellite communication unit 61. Then, in step S42, the control unit 63 performs imaging to generate a high-resolution image with the normal performance of imaging device 62. For example, imaging device 62 performs imaging to generate 4K resolution when it can perform imaging at the maximum 4K resolution, and performs imaging to generate HD resolution when it can perform imaging at the maximum HD resolution. The image data of the high-resolution image obtained by imaging is stored in storage unit 64. The image data of the high-resolution image can be stored in storage unit 64 as raw data or as encoded data encoded by a predetermined encoding.
[0060] In step S43, the control unit 63 performs a capacity reduction process to generate small-capacity data with a capacity smaller than that of the high-resolution image, based on the high-resolution image obtained by imaging.
[0061] In the capacity reduction process, for example, information representing the satellite's state when satellite 21 performs imaging is generated as small-capacity data. This information includes, for example, attitude information representing the satellite 21's attitude when performing imaging, and setting information (hereinafter referred to as camera setting information) when imaging device 62 performs imaging. Camera setting information includes, for example, information related to camera settings such as resolution, zoom ratio, shutter speed, sensitivity, and aperture. When imaging device 62 is attached to satellite 21 at a fixed position, the imaging range (angle of view) of imaging device 62 can be specified based on the satellite 21's attitude and the imaging device 62's zoom setting. Therefore, the satellite 21's attitude information can also be considered part of the camera setting information.
[0062] In file size reduction processing, low-capacity images can be generated as small-capacity data by converting high-resolution images obtained through imaging into low-capacity images. For example, low-resolution images obtained by converting high-resolution images to low resolution (e.g., VGA), panchromatic images obtained by converting high-resolution color images to monochrome images, image feature information obtained by extracting feature quantities of characteristic subjects included in high-resolution images, and partial images obtained by cropping only a portion of a high-resolution image can be generated as small-capacity data. Examples of image feature information obtained by extracting feature quantities from high-resolution images include, for example, information representing cloud shape characteristics, information about moving subjects (e.g., airplanes), information about light hues (e.g., red sunsets), and information about the location of ground reflections caused by sunlight (e.g., the reflection state of oceans or lakes).
[0063] In the capacity reduction process, a low frame rate image, obtained by converting the frame rate of a high-resolution image acquired through imaging to a lower frame rate, can be generated as small-capacity data. For example, the control unit 63 generates a low frame rate image, obtained by converting the frame rate of a high-resolution image at 30 fps to a frame rate of 1 fps, as small-capacity data.
[0064] Small-capacity data is generated by performing a capacity reduction process on information about the current imaging of satellite 21.
[0065] In step S44, the control unit 63 transmits the small-capacity data generated through the capacity reduction process to the satellite management device 11 via the satellite communication unit 61. The satellite 21 repeats the above steps S42 to S44 until an imaging end indication is sent from the satellite management device 11. By repeating steps S42 to S44, the small-capacity data is transmitted to the satellite management device 11 at a predetermined frame rate. Because the small-capacity data has a small capacity, it can be transmitted in frames without delay.
[0066] In step S12, the control unit 81 of the satellite management device 11 receives a small amount of data transmitted from the satellite 21 via the communication unit 82. In step S13, supplementary processing to supplement the small amount of data is performed to generate a live view image. Then, in step S14, the control unit 81 causes the display unit 85 to display the generated live view image.
[0067] When performing supplementary processing, the control unit 81 retrieves relevant information from one or more information providing servers 41 of an external organization as needed. For example, the relevant information includes: For instance, the control unit 81 may retrieve TLE information as relevant information and specify the position of satellite 21 at the time of imaging. For instance, the control unit 81 retrieves archived images (baseline images) and near-real-time images as relevant information. The archived images (baseline images) are satellite images of the same imaging location accumulated in the past (e.g., a few days or a month ago) by an operating company that operates other satellite observation services, while the near-real-time images are satellite images of the same imaging location at a relatively recent time, tens of minutes to several hours before the current time. Of course, if similar images are stored in its own (satellite management device 11) storage unit 83, rather than by an operating company that operates other satellite observation services, then those images can be used. Preferably, the archived images or near-real-time images to be acquired have the same season, imaging time, weather conditions, etc., as when the small volume of data was generated. For example, the control unit 81 may retrieve meteorological information, such as weather, cloud distribution, cloud cover, and sun position information, from a meteorological information providing company that is an external organization at the time the small volume of data was generated. For example, control unit 81 can obtain AIS (Automatic Identification System) information indicating the position of the ship and aircraft at the time of imaging from information providing server 41 of the operation information providing company that provides operational information of ships and aircraft as relevant information.
[0068] The control unit 81 performs supplementary processing using the aforementioned relevant information as needed, and generates a real-time view image (satellite image) estimated from the small amount of data captured by the satellite 21.
[0069] For example, when the small amount of data is information representing the satellite's status (such as the attitude information and camera setting information of satellite 21), the control unit 81 uses TLE information, baseline images, near real-time images, etc. as relevant information, and reproduces the images captured by the imaging device 62 from the perspective captured by the imaging device 62 through CG (computer graphics), etc., to generate a real-time view image.
[0070] For example, when a small amount of data is obtained by converting a high-resolution image to a low-resolution image (e.g., VGA), the control unit 81 uses super-resolution technology to generate a high-resolution image from the low-resolution image to obtain a real-time view image. At this time, a baseline image or a near-real-time image can be acquired as relevant information, and interpolation processing can be performed on it.
[0071] For example, when the small amount of data is a panchromatic image obtained by converting a high-resolution color image into a monochrome image, the control unit 81 uses machine learning to generate a high-resolution color image from the panchromatic image to obtain a real-time view image. In this case, image feature information extracted from the features included in the high-resolution image can be used to perform colorization processing.
[0072] For example, when the small amount of data is image feature information obtained by extracting features included in a high-resolution image, the control unit 81 reproduces the feature subject based on information about cloud shape, dynamic subjects, and light hue, as well as information about the location of sunlight reflection, which are image feature information. The feature subject is then overlaid on an archived image or a near-real-time image to generate a real-time view image. For example, when acquiring a panchromatic image and image feature information as small amount of data, processing such as adding light hue to the panchromatic image or overlaying cloud shape and dynamic subjects onto the panchromatic image can be performed. External information can also be used for meteorological information and the location of dynamic subjects.
[0073] For example, when the small amount of data is a low frame rate image, the control unit 81 generates a live view image by interpolating between frames of the acquired low frame rate image and generating an image to increase the frame rate. The frame rate of the live view image is not necessarily the same as the frame rate of the high-resolution image.
[0074] The supplementary processing in step S13 can be omitted. For example, when the small amount of data is a low-resolution image obtained by converting a high-resolution image to a low-resolution image, or a panchromatic image obtained by converting a high-resolution image to a monochrome image, the supplementary processing can be omitted, and the low-resolution image or panchromatic image can be displayed as is on the display unit 85 as a live view image.
[0075] In the display of the live view image in step S14, the control unit 81 can display the live view image displayed on the display unit 85 so that the user can understand that the live view image is an image estimated based on a small amount of data. For example, information (characters) indicating that it is an estimated image can be superimposed on the live view image, or an outline image indicating that it is an estimated image can be added to and displayed on the outer periphery of the live view image.
[0076] In step S15, the control unit 81 determines whether the user's operation on the satellite 21 has been executed by the operation unit 84. If it is determined in step S15 that the user's operation has not been executed, the process returns to step S12, and the above-described steps S12 to S15 are repeated.
[0077] On the other hand, when it is determined in step S15 that the user's operation has been performed, the process proceeds to step S16, and the control unit 81 sends a control instruction corresponding to the user's operation to the satellite 21 via the communication unit 82.
[0078] Examples of user operations in step S15 include the following: For example, the user may perform an operation instructing the user to change camera settings (e.g., resolution, zoom ratio, shutter speed, sensitivity, and aperture). For example, the user may perform an operation to change the imaging direction (imaging location) of the imaging device 62. When the imaging device 62 is attached to the satellite 21 in a fixed position, the attitude of the satellite 21 itself is changed in response to the instruction to change the imaging direction. When the imaging device 62 has a movable portion capable of changing its relative position with respect to the satellite 21, the attitude of the satellite 21 itself may be changed in response to the instruction to change the imaging direction, or the relative position of the imaging device 62 may be changed. The zoom ratio may be optical zoom or digital zoom.
[0079] For example, as a user operation in step S15, the user can give an instruction to request a large amount of live view image data, which is a high-resolution image corresponding to the live view image displayed on the display unit 85.
[0080] Corresponding to step S16 of the satellite management device 11, in step S45, the control unit 63 of the satellite 21 receives a control instruction sent from the satellite management device 11 via the satellite communication unit 61.
[0081] Subsequently, in step S46, the control unit 63 determines whether the control instruction from the satellite management device 11 is a large-capacity data transmission instruction.
[0082] When it is determined in step S46 that the control instruction from satellite management device 11 is not a mass data transmission instruction, the process proceeds to step S47, and control unit 63 performs control based on the control instruction. For example, when the control instruction is a change in camera setting information such as resolution and zoom ratio, control unit 63 performs control to change a portion of the camera setting information of imaging device 62 to a specified setting value. After the processing in step S47, the camera setting values of imaging device 62 for capturing high-resolution images, which have been repeatedly executed in steps S42 to S44 above, are changed.
[0083] As a general control procedure, after changing the camera settings of the imaging device 62 in response to a control instruction from the satellite management device 11, high-resolution image capture begins with the changed camera settings. Therefore, there is a slight delay before the user's camera setting information change instruction is reflected in the live view image.
[0084] However, under certain imaging conditions and the following combination of camera setting information change indications, the delay perceived by the user when checking the live view image can be reduced.
[0085] For example, in the case of capturing a high-resolution image in step S42 above, when the imaging device 62 changes the predetermined camera setting value to multiple values and performs bracket shooting using multiple different camera setting values, that is, when the imaging device 62 performs bracket shooting using camera setting information changed by user instruction, a small amount of data based on the high-resolution image captured using the camera setting value changed by user instruction can be immediately sent to the satellite management device 11.
[0086] For example, when the data sent as small-capacity data is partial image data that only cuts out a portion of a high-resolution image, and the user's camera setting information change instruction is to change the zoom ratio or viewpoint, the high-resolution image before the cutout can be used to generate a partial image with the changed zoom ratio and viewpoint, and this can be sent immediately as small-capacity data to the satellite management device 11.
[0087] refer to Figure 4 The description will be an example of how changing the user's camera settings is an example of changing the viewpoint.
[0088] At time t1, the user issues a camera setting information change instruction so that while viewing the partial image C1 obtained by only cropping the central part of the high-resolution image B1 on the display unit 85, the viewing angle is moved to the right and the partial image C1 is viewed simultaneously.
[0089] At time t2, the control unit 63 of satellite 21 receives a camera setting information change instruction and begins attitude control to change the attitude of imaging device 62 in the indicated direction. However, since the attitude of imaging device 62 does not immediately change to the desired attitude, the viewing angle of the high-resolution image B2 captured by imaging device 62 at time t2 is substantially the same as that of the high-resolution image B1 at time t1. The control unit 63 cuts out the portion of image C1 corresponding to the viewing angle indicated by the user from the high-resolution image B2 captured by imaging device 62 at time t2 and immediately sends it as a small amount of data to satellite management device 11.
[0090] At time t3, when the orientation of the imaging device 62 is aligned with the imaging direction specified by the user, a portion of the image C3 obtained by cropping only the central part of the high-resolution image B3 is sent as a small amount of data to the satellite management device 11.
[0091] As described above, since a portion of the image with a user-specified viewpoint can be displayed on the display unit 85 before the orientation of the imaging device 62 reaches a state that has been changed by the user, the delay perceived by the user when checking the live view image can be reduced.
[0092] Return to Figure 3 The flowchart shows that when it is determined in step S46 that the control instruction from the satellite management device 11 is a mass data transmission instruction, the process proceeds to step S48, and the control unit 63 transmits the indicated mass data to the satellite management device 11 via the satellite communication unit 61. That is, data of a high-resolution image corresponding to the real-time view image displayed on the display unit 85 in the process of step S14 is transmitted from the satellite 21 to the satellite management device 11.
[0093] In satellite management device 11, after sending the control instruction corresponding to the user's operation to satellite 21 in step S16, the process proceeds to step S17, and control unit 81 determines whether the control instruction sent to satellite 21 is a large-capacity data transmission instruction.
[0094] When it is determined in step S17 that the control instruction sent to satellite 21 is not a large data transmission instruction, the process returns to step S12 and the process after step S12 is executed again.
[0095] On the other hand, when it is determined in step S17 that the control instruction sent to satellite 21 is a large data transmission instruction, the process proceeds to step S18, and control unit 81 receives the large data transmitted from satellite 21 in response to the sent control instruction and stores it in storage unit 83. After step S18, the process returns to step S12, and the processing after step S12 is executed again.
[0096] Continue processing the first live view image display until the user executes the live view image display end operation, and the first live view image display processing ends when the live view image display end operation is executed.
[0097] In the first real-time view image display process, the satellite management device 11 receives small-capacity data from the satellite 21, the size of which is smaller than the capacity of a high-resolution image with a resolution that is within the normal performance range of the imaging device 62. Because the small-capacity data has a small capacity, it can be transmitted frame by frame without delay, thus enabling the real-time view image based on the small-capacity data to be displayed in real time. Then, based on an instruction from a user who has checked the real-time view image displayed on the display unit 85, large-capacity data is received and stored as a high-resolution image corresponding to the real-time view image.
[0098] The transmission and reception of large amounts of data processed in steps S18 and S48 above can be performed at a time other than during the execution of the first real-time viewfinder image display process described above; in other words, at a time other than the time when the real-time viewfinder image is observed.
[0099] When there are two or more communication systems between satellite 21 and communication equipment 13, the transmission and reception of large-capacity data can be performed in a different background than the transmission and reception of small-capacity data. In this case, because large-capacity data requires a longer communication time, the number of times large-capacity data is transmitted is less than the number of times small-capacity data is transmitted. For example, when small-capacity data is transmitted at 30 fps, large-capacity data is transmitted at 1 fps or every 10 minutes. In the supplementary processing of step S13 above, the large-capacity data transmitted and received in the background can be used as near-real-time imagery.
[0100] Large volumes of data can be used as teacher data when machine learning is performed as supplementary processing for estimating live view images. Using machine learning to generate live view images can improve the accuracy of live view images generated by estimation.
[0101] <3. Second Real-Time View Image Display Processing>
[0102] Next, the second real-time viewfinder image display processing performed by the satellite image processing system 1 will be described.
[0103] In the aforementioned first real-time image display process, satellite 21 captures a high-resolution image based on an imaging start instruction from satellite management device 11, generates a small amount of data based on the high-resolution image, and sends it to satellite management device 11. In this case, the high-resolution image is stored in storage unit 64 and, as needed, is sent to satellite management device 11 as a large amount of data at other time intervals.
[0104] On the other hand, in the second live view image display processing, satellite 21 does not capture high-resolution images, but instead performs imaging to generate small-volume data. Then, only when the user observing the live view image gives an instruction to capture a high-resolution image, satellite 21 captures a high-resolution image corresponding to the live view image and transmits it as large-volume data.
[0105] Reference Figure 5 The flowchart describes the second live view image display process. For example, this process begins when a user operating the satellite management device 11 performs a live view image display start operation.
[0106] First, in step S61, the control unit 81 of the satellite management device 11 sends an imaging start instruction to the satellite 21 via the communication unit 82.
[0107] In step S81, the control unit 63 of satellite 21 receives an imaging start instruction sent from satellite management equipment 11 via satellite communication unit 61. Then, in step S82, the control unit 63 performs small-volume data generation processing.
[0108] In the small-capacity data generation process, the same small-capacity data as the first real-time image display process described above is generated. For example, information representing the satellite state when satellite 21 performs imaging is generated as small-capacity data. Alternatively, a low-capacity image with a smaller capacity than a high-resolution image captured using the normal performance of imaging device 62 (e.g., a low-resolution image with a lower resolution than a color high-resolution image, a panchromatic image obtained by converting a color high-resolution image into a monochrome image, image feature quantity information obtained by extracting feature quantities of characteristic subjects included in a high-resolution image, a partial image corresponding to a partial region of a high-resolution image, and a low frame rate image with a frame rate lower than that of a high-resolution image) is generated as small-capacity data.
[0109] In step S83, the control unit 63 transmits the small-capacity data generated through the small-capacity data generation process to the satellite management device 11 via the satellite communication unit 61. The satellite 21 repeats steps S82 and S83 until an imaging end indication is sent from the satellite management device 11. By repeating steps S82 and S83, the small-capacity data is transmitted to the satellite management device 11 at a predetermined frame rate. Because the small-capacity data has a small capacity, it can be transmitted in frames without delay.
[0110] In step S62, the control unit 81 of the satellite management device 11 receives a small amount of data transmitted from the satellite 21 via the communication unit 82. In step S63, supplementary processing to supplement the small amount of data is performed to generate a live view image. Then, in step S64, the control unit 81 causes the display unit 85 to display the generated live view image.
[0111] The details of the supplementary processing in step S63 and the display of the live view image in step S64 are the same as those of the first live view image display processing described above, so their description will be omitted. Furthermore, the supplementary processing in step S63 can be omitted.
[0112] In step S65, the control unit 81 determines whether the user's operation on the satellite 21 has been executed by the operation unit 84. If it is determined in step S65 that the user's operation has not been executed, the process returns to step S62, and the above-described processes from S62 to S65 are repeated.
[0113] On the other hand, when it is determined in step S65 that the user's operation has been performed, the process proceeds to step S66, and the control unit 81 sends a control instruction corresponding to the user's operation to the satellite 21 via the communication unit 82. The operations that the user can perform while observing the live view image are the same as the operations described in the first live view image display process.
[0114] Corresponding to step S66 of the satellite management device 11, in step S84, the control unit 63 of the satellite 21 receives a control instruction sent from the satellite management device 11 via the satellite communication unit 61.
[0115] Subsequently, in step S85, the control unit 63 determines whether the control instruction from the satellite management device 11 is a large-capacity data transmission instruction.
[0116] When it is determined in step S85 that the control instruction from satellite management device 11 is not a large-capacity data transmission instruction, the process proceeds to step S86, and control unit 63 performs control based on the control instruction. For example, when the control instruction is a change in camera setting information such as resolution and zoom ratio, control unit 63 performs control to change a portion of the camera setting information of imaging device 62 to a specified setting value. After the processing in step S86, the camera setting values of imaging device 62 used to generate small-capacity data, which are repeatedly executed in steps S82 and S83 above, are changed.
[0117] On the other hand, when it is determined in step S85 that the control instruction from the satellite management device 11 is a mass data transmission instruction, the process proceeds to step S87, and the control unit 63 performs imaging to generate a high-resolution image with the normal resolution of the imaging device 62. The generated high-resolution image is stored in the storage unit 64. Subsequently, in step S88, the control unit 63 transmits the image data of the generated high-resolution image as mass data to the satellite management device 11 via the satellite communication unit 61.
[0118] In satellite management device 11, after sending the control instruction corresponding to the user's operation to satellite 21 in step S66, the process proceeds to step S67, and control unit 81 determines whether the control instruction sent to satellite 21 is a large-capacity data transmission instruction.
[0119] When it is determined in step S67 that the control instruction sent to satellite 21 is not a large data transmission instruction, the process returns to step S62, and the process after step S62 is executed again.
[0120] On the other hand, when it is determined in step S67 that the control instruction sent to satellite 21 is a large-capacity data transmission instruction, the process proceeds to step S68, and the control unit 81 receives the large-capacity data transmitted from satellite 21 according to the sent control instruction and stores it in the storage unit 83. After step S68, the process returns to step S62, and the processing after step S62 is executed again.
[0121] Continue executing the second live view image display process until the user executes the live view image display end operation, and the second live view image display process ends when the live view image display end operation is executed.
[0122] Similar to the first real-time view image display process, the transmission and reception of large-capacity data, which are part of the processes described in steps S68 and S88, can be performed at a time other than during the execution of the second real-time view image display process described above. In other words, it can be performed at a time other than during the real-time observation of the real-time view image.
[0123] Based on the first and second real-time image display processing performed by the satellite image processing system 1, small amounts of data are transmitted while observing the real-time image, and high-resolution images are transmitted as large amounts of data to the satellite management device 11 at other time intervals as needed. In this way, users can check images on the ground in real time.
[0124] <4. Application Examples>
[0125] This technology can be applied not only to satellite image processing systems that remotely direct imaging via artificial satellites, but also to the remote control and monitoring of mobile systems operating in remote areas.
[0126] Mobile systems are systems that include, for example, robots, particularly teleexistence robots, autonomous vehicles, autonomous aircraft (drones), manned vehicles, or manned aircraft as mobile devices. In these mobile systems, users operating and monitoring the mobile devices from remote locations can use real-time viewfinder images with a small amount of data to check a minimal situation.
[0127] As in the case of satellite image processing system 1 described above, a real-time view image can be generated by performing supplementary processing to supplement small amounts of data.
[0128] For example, a hypothetical image can be reconstructed (estimated) based on the mobile device's pose information, location information, etc. In this case, it can be supplemented when the mobile device's movement path and environment are known, or when archived images of the same location in the past or near-real-time images of the same location taken at a relatively recent time are accumulated to provide information about the movement path and environment.
[0129] VGA image-based super-resolution techniques can be used to generate high-resolution live view images, generate high-resolution color images from panchromatic images, and interpolate low-frame-rate images to generate high-frame-rate live view images.
[0130] Real-time images can also be generated based on feature information. For example, when the movement path and environment of the mobile device are known, or when information about the movement path and environment is accumulated, real-time features of only the subject unique to the movement path and environment can be extracted from archived images to generate an estimated image.
[0131] <5. Computer Configuration Example>
[0132] The series of processes described above, performed by satellite management device 11, can also be executed by hardware or software. When the series of processes are executed by software, a program including that software is installed in a computer. Here, the computer includes a computer embedded in dedicated hardware, or, for example, a general-purpose personal computer capable of performing various functions by installing various programs.
[0133] Figure 6 This is a block diagram illustrating an example configuration of the hardware of a computer that performs a series of processes by the satellite management device 11 through a program.
[0134] In a computer, the central processing unit (CPU) 301, read-only memory (ROM) 302, and random access memory (RAM) 303 are connected to each other via bus 304.
[0135] The input / output interface 305 is further connected to the bus 304. The input unit 306, output unit 307, storage unit 308, communication unit 309, and driver 310 are connected to the input / output interface 305.
[0136] Input unit 306 is, for example, a keyboard, mouse, microphone, touch panel, or input terminal. Output unit 307 is, for example, a display, speaker, or output terminal. Storage unit 308 is, for example, a hard disk, RAM disk, or non-volatile memory. Communication unit 309 is a network interface, etc. Driver 310 drives removable recording medium 311, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.
[0137] In a computer with the above configuration, the CPU 301 performs the aforementioned series of processes, for example, by loading a program stored in the storage unit 308 into the RAM 303 via the input / output interface 305 and the bus 304 and executing the program. The RAM 303 also appropriately stores data required for the CPU 301 to perform various processes.
[0138] The program executed by the computer (CPU 301) can be recorded on a removable recording medium 311, such as a packaging medium used for supply. The program can be provided via wired or wireless transmission media, such as a local area network, the Internet, or digital satellite broadcasting.
[0139] In a computer, a program can be installed in a storage unit 308 via an input / output interface 305 by mounting a removable recording medium 311 on a drive 310. A communication unit 309 can receive the program via a wired or wireless transmission medium for installation in the storage unit 308. Alternatively, the program can be pre-installed in a ROM 302 or in the storage unit 308.
[0140] In this specification, even if these steps are not performed in the order described in the flowchart in time sequence, and when these steps are performed in time sequence, the steps described in the flowchart may be performed in parallel or at the required time intervals (e.g., when invoked).
[0141] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), and all components may or may not be housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a device in which multiple modules are housed in one housing, can both be considered a system.
[0142] The embodiments of this technology are not limited to the above embodiments, and various changes can be made without departing from the essential points of this technology.
[0143] For example, a combination of all or part of the above embodiments may be used.
[0144] For example, this technology can have a cloud computing configuration, in which multiple devices share and work together to perform a function via a network.
[0145] Furthermore, each step described in the flowchart above can be performed by one device or shared by multiple devices.
[0146] Furthermore, in cases where a step includes multiple processes, the multiple processes included in a step can be executed by a single device, or can be shared and executed by multiple devices.
[0147] The effects described in this specification are merely illustrative and not limited, and effects other than those described in this specification may exist.
[0148] This technology can be configured as follows.
[0149] (1) A display control device, comprising:
[0150] A receiving unit receives a small amount of data as information about the current imaging of an artificial satellite; and
[0151] The control unit displays a real-time viewfinder image based on a small amount of data on the display unit, wherein...
[0152] The receiving unit receives satellite images corresponding to real-time view images as large-capacity data at a time different from that of small-capacity data.
[0153] (2) The display control device according to (1), wherein,
[0154] The receiving unit receives large amounts of data transmitted from satellites based on instructions from a user who has checked the live view image displayed on the display unit.
[0155] (3) The display control device according to (1) or (2), wherein,
[0156] The satellite generates smaller datasets by performing a file reduction process that decreases the size of the captured satellite images.
[0157] Large-capacity data refers to satellite images before they undergo capacity reduction processing.
[0158] (4) The display control device according to (1) or (2), wherein,
[0159] The receiving unit receives satellite images captured by artificial satellites as large-capacity data based on instructions from a user who has verified the live view image displayed on the display unit.
[0160] (5) The display control device according to (1), wherein,
[0161] Small-capacity data refers to information indicating the satellite's status.
[0162] (6) The display control device according to (5), wherein,
[0163] The information indicating the satellite's status includes attitude information indicating the satellite's orientation and camera settings information when the satellite performs imaging.
[0164] (7) The display control device according to (1), wherein,
[0165] Small-capacity data is a low-capacity image obtained by converting satellite images acquired through imaging of the satellite into images with small capacity.
[0166] (8) The display control device according to (7), wherein,
[0167] The control unit displays the low-capacity image as a live view image on the display unit as is.
[0168] (9) The display control device according to (1), (7) or (8), wherein,
[0169] Small-capacity data is low-resolution images obtained by converting satellite images acquired through imaging of the satellite into images with low resolution.
[0170] (10) The display control device according to (1), (7) or (8), wherein,
[0171] Small-capacity data is obtained by converting satellite images acquired through imaging of the artificial satellite into monochrome images, resulting in panchromatic images.
[0172] (11) The display control device according to (1), wherein,
[0173] Small-capacity data is image feature information obtained by extracting the feature quantities of featured subjects included in satellite images obtained through imaging of the artificial satellite.
[0174] (12) The display control device according to (1), (7) or (8), wherein,
[0175] Small-capacity data is obtained by cropping only a portion of the satellite imagery obtained through imaging of the satellite.
[0176] (13) The display control device according to (1), (7) or (8), wherein,
[0177] Small-capacity data is low-frame-rate images obtained by converting the frame rate of satellite images acquired through imaging of the satellite to a lower frame rate than that of the satellite images.
[0178] (14) The display control device according to any one of (1) to (13), wherein,
[0179] The control unit performs supplementary processing to supplement small amounts of data, generates a real-time view image, and displays the real-time view image on the display unit.
[0180] (15) The display control device according to (14), wherein,
[0181] The control unit uses past images obtained by imaging the same location to generate a real-time view image when performing supplementary processing.
[0182] (16) The display control device according to (14), wherein,
[0183] Small-capacity data refers to low-resolution images obtained by converting satellite images acquired through imaging of the satellite into images with lower resolution, or low-frame-rate images obtained by converting the frame rate of satellite images acquired through imaging of the satellite to a frame rate lower than that of satellite images.
[0184] The control unit increases the resolution or frame rate to generate a real-time view image as a supplementary process.
[0185] (17) According to (14),
[0186] Small-volume data is image feature information obtained by extracting the feature quantities of characteristic subjects included in satellite images obtained through imaging of the artificial satellite, and
[0187] The control unit uses image feature information to estimate and generate real-time view images as supplementary processing.
[0188] (18) The display control device according to (14), wherein,
[0189] Small-capacity data refers to panchromatic images obtained by converting satellite images acquired through imaging of the aforementioned satellite into monochrome images, and...
[0190] The control unit generates a real-time color viewfinder image based on the panchromatic image as supplementary processing.
[0191] (19) A display control method for allowing a display control device to perform:
[0192] Receive small amounts of data as information about current imaging of artificial satellites;
[0193] Displaying a live view image based on a small amount of data on the display unit; and
[0194] Unlike small-capacity data, large-capacity data is generated by receiving satellite images corresponding to real-time viewfinder images at different times.
[0195] (20) A program that causes a computer to execute:
[0196] Receive small amounts of data as information about current imaging of artificial satellites;
[0197] Displaying a live view image based on a small amount of data on the display unit; and
[0198] Unlike small-capacity data, large-capacity data is generated by receiving satellite images corresponding to real-time viewfinder images at different times.
[0199] Reference tag list
[0200] 1. Satellite Image Processing System
[0201] 11 Satellite Management Equipment
[0202] 13. Communication equipment
[0203] 15 Ground Stations
[0204] 21 satellites
[0205] 41 Information Provider Server
[0206] 42 Image Analysis Servers
[0207] 61 Satellite Communication Unit
[0208] 62 Imaging equipment
[0209] 63 Control Unit
[0210] 64 storage units
[0211] 81 Control Unit
[0212] 82 Communication Units
[0213] 83 storage units
[0214] 84 operating units
[0215] 85 display units
[0216] 301 CPU
[0217] 302 ROM
[0218] 303 RAM
[0219] 306 Input Unit
[0220] 307 Output Unit
[0221] 308 memory cells
[0222] 309 Communication Unit
[0223] 310 drive
Claims
1. A display control device, comprising: A receiving unit receives small-volume data as information about current imaging of an artificial satellite and relevant information about the imaging location, wherein the relevant information includes at least one of the following: past images obtained by imaging the same imaging location, or meteorological information about the same imaging location at the time the small-volume data was generated; and The control unit displays a real-time view image on a display unit based on a small amount of data and the relevant information, wherein... The receiving unit receives satellite images corresponding to real-time view images as large-capacity data at a time different from that of small-capacity data. Small-volume data includes information representing the satellite's status; Information indicating the satellite's state includes attitude information representing the satellite's orientation and camera setting information when the satellite performs imaging; and The control unit generates a real-time view image of a satellite image captured by the satellite by using the attitude information, the camera setting information, and the relevant information to estimate the perspective captured by the satellite.
2. The display control device according to claim 1, wherein, The receiving unit receives large amounts of data transmitted from satellites based on instructions from a user who has checked the live view image displayed on the display unit.
3. The display control device according to claim 1, wherein, The satellite generates smaller datasets by performing a file-reduction process that reduces the size of the captured satellite images. Large-capacity data refers to satellite images before they undergo capacity reduction processing.
4. The display control device according to claim 1, wherein, The receiving unit receives satellite images captured by artificial satellites as large-capacity data based on instructions from a user who has verified the live view image displayed on the display unit.
5. The display control device according to claim 1, wherein, Small-capacity data includes low-capacity images obtained by converting satellite images acquired through imaging of the satellite into images with small capacities.
6. The display control device according to claim 5, wherein, The control unit displays the low-capacity image as a live view image on the display unit as is.
7. The display control device according to claim 1, wherein, Small-capacity data includes low-resolution images obtained by converting satellite images acquired through imaging of the satellite into images with low resolution.
8. The display control device according to claim 1, wherein, Small-capacity data includes panchromatic images obtained by converting satellite images acquired through imaging of the satellite into monochrome images.
9. The display control device according to claim 1, wherein, Small-volume data includes image feature information obtained by extracting feature quantities of characteristic subjects included in satellite images obtained through imaging of the said satellite.
10. The display control device according to claim 1, wherein, Small-capacity data includes partial images obtained by cropping only a portion of a satellite image acquired through imaging of the satellite.
11. The display control device according to claim 1, wherein, Small-capacity data includes low-frame-rate images obtained by converting the frame rate of satellite images acquired through imaging of the satellite to a lower frame rate than that of the satellite images.
12. The display control device according to claim 1, wherein, The control unit performs supplementary processing to supplement small amounts of data, generates a real-time view image, and displays the real-time view image on the display unit.
13. The display control device according to claim 12, wherein, Small-capacity data includes low-resolution images obtained by converting satellite images acquired through imaging of the satellite into images with lower resolution, or low-frame-rate images obtained by converting the frame rate of satellite images acquired through imaging of the satellite to a frame rate lower than that of the satellite images, and... The control unit increases the resolution or frame rate to generate a real-time view image as a supplementary process.
14. The display control device according to claim 12, wherein, Small-volume data includes image feature information obtained by extracting feature quantities of characteristic subjects included in satellite images obtained through imaging of the said satellite, and The control unit uses image feature information to estimate and generate real-time view images as supplementary processing.
15. The display control device according to claim 12, wherein, Small-capacity data includes panchromatic images obtained by converting satellite images acquired through imaging of the satellite into monochrome images, and... The control unit generates a real-time color viewfinder image based on the panchromatic image as supplementary processing.
16. A display control method for allowing a display control device to perform: Receive a small volume of data as information about the current imaging of the satellite and relevant information about the imaging location, wherein, The relevant information includes at least one of the following: past images obtained by imaging the same imaging location, or meteorological information of the same imaging location when generating small amounts of data; A real-time view image based on a small amount of data and the related information is displayed on the display unit; as well as Unlike small-capacity data, large-capacity data is generated by receiving satellite images corresponding to real-time viewfinder images at different times. Small-volume data includes information representing the satellite's status; Information indicating the satellite's state includes attitude information representing the satellite's orientation and camera setting information when the satellite performs imaging; and A real-time view image is generated by estimating the perspective captured by the satellite using the attitude information, the camera setting information, and the relevant information.
17. A program product that causes a computer to execute: Receive a small volume of data as information about the current imaging of the satellite and relevant information about the imaging location, wherein, The relevant information includes at least one of the following: past images obtained by imaging the same imaging location, or meteorological information of the same imaging location when generating small amounts of data; A real-time view image based on a small amount of data and the related information is displayed on the display unit; as well as Unlike small-capacity data, large-capacity data is generated by receiving satellite images corresponding to real-time viewfinder images at different times. Small-volume data includes information representing the satellite's status; Information indicating the satellite's state includes attitude information representing the satellite's orientation and camera setting information when the satellite performs imaging; and A real-time view image is generated by estimating the perspective captured by the satellite using the attitude information, the camera setting information, and the relevant information.
Citation Information
Patent Citations
Data compressor and data expanding device
JP2006115283A
Earth observation method
US5883584A