Image processing apparatus, image processing system, image processing method, and storage medium

Abstract

To determine a position in a depth direction of a subject expressed in image data captured and generated by an existing imaging apparatus without adding an imaging device.SOLUTION: Acquiring image data generated by imaging by imaging means arranged to have different line-of-sight directions, performing alignment so as to eliminate an absolute positional deviation caused by the different line-of-sight directions in each of the acquired image data, and superimposing the image data; The image processing apparatus selects a transversal line crossing an overlapping part of the superimposed image data according to a direction in which a range of each of the superimposed image data is shifted, generates epipolar plane image data by arranging each of image data of a part where the selected transversal line and the overlapping part overlap in an order determined by a magnitude of an inclination of a line-of-sight direction corresponding to each of the image data, and detects a depth of an object expressed in the image data from a streak pattern expressed in the generated epipolar plane image data.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an image processing system, an image processing method, and a program. [Background technology]

[0002] As disclosed in Patent Document 1, clouds in satellite images are identified using, for example, reflection intensity, which is the pixel value of a pixel in a visible image, or brightness temperature, which is the pixel value of a pixel in an infrared image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-086449 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the visible light wavelength band, both ice and snow and clouds have high and similar reflectances of sunlight, resulting in similar reflection intensities, making it difficult to distinguish between them using the method disclosed in Patent Document 1.

[0005] In contrast, in the infrared wavelength band, differences in reflectance occur between ice and snow and clouds, which in turn results in differences in brightness temperature, making it possible to distinguish between ice and snow and clouds using the method disclosed in Patent Document 1. Furthermore, by using an active sensor such as LiDAR (Light Detection and Ranging), it is possible to detect the position of an object in the depth direction by irradiating the object with a signal and measuring the distance, making it possible to distinguish between ice and snow on the ground and clouds floating in the air.

[0006] However, in order to use devices such as infrared sensors and LiDAR that can receive infrared wavelength bands, these devices must be added to the satellite along with existing imaging devices that capture and generate satellite images. However, this addition increases costs due to the additional devices and the cost of launching the satellite due to the increased weight of the satellite. Therefore, if you want to suppress the increase in costs, you will encounter the problem of how to determine the depth position of a subject represented in image data captured and generated by an existing imaging device without adding any imaging devices.

[0007] An object of the present disclosure is to provide an image processing device, an image processing system, an image processing method, and a program that solve the above-mentioned problems. [Means for solving the problem]

[0008] An image processing device according to one aspect of the present disclosure includes an image acquisition means for acquiring image data generated by imaging means arranged so as to have different line of sight directions; an image superimposition means for aligning and superimposing the image data so as to eliminate absolute positional shifts that arise due to the different line of sight directions in each of the image data; an epipolar plane image generation means for selecting a transverse line that crosses the overlapping portion of the superimposed image data according to the direction in which the ranges of each of the image data in the superimposed state shift, and arranging each of the image data in the portion where the selected transverse line and the overlapping portion overlap in an order determined by the magnitude of the inclination of the line of sight direction corresponding to each of them, to generate epipolar plane image data; and a depth detection means for detecting the depth of the subject represented in the image data from the streak pattern represented in the epipolar plane image data.

[0009] An image processing system according to one aspect of the present disclosure includes an imaging means and an image processing device, and the image processing device includes: an image acquisition means for acquiring image data generated by imaging using the imaging means arranged so as to have different line of sight; an image superimposition means for aligning and superimposing the image data to eliminate absolute positional shifts that arise due to the different line of sight directions in each of the image data; an epipolar plane image generation means for selecting a transverse line that crosses the overlapping portion of the superimposed image data according to the direction in which the ranges of each of the image data in the superimposed state shift, and arranging each of the image data in the portion where the selected transverse line and the overlapping portion overlap in an order determined by the magnitude of the inclination of the line of sight direction corresponding to each of the selected transverse line and the overlapping portion to generate epipolar plane image data; and a depth detection means for detecting the depth of the subject represented in the image data from the streak pattern represented in the epipolar plane image data.

[0010] An image processing method according to one aspect of the present disclosure acquires image data generated by imaging means positioned so as to have different line-of-sight directions, aligns the image data to eliminate absolute positional shifts that arise from the different line-of-sight directions in each of the acquired image data, and overlays the image data, selects a transverse line that crosses the overlapping portion of the overlaid image data according to the direction in which the ranges of each of the overlaid image data shift, arranges each of the image data in the portion where the selected transverse line and the overlapping portion overlap in an order determined by the magnitude of the inclination of the corresponding line-of-sight direction to generate epipolar plane image data, and detects the depth of the subject represented in the image data from the streak pattern represented in the generated epipolar plane image data.

[0011] A program according to one aspect of the present disclosure causes a computer to function as an image acquisition means for acquiring image data generated by imaging means positioned so as to have different line-of-sight directions; an image superimposition means for aligning the image data to eliminate absolute positional shifts that arise due to the different line-of-sight directions in each of the image data and superimposing the image data; an epipolar plane image generation means for selecting a transverse line that crosses the overlapping portion of the superimposed image data according to the direction in which the range of each of the image data in the superimposed state shifts, and arranging each of the image data in the portion where the selected transverse line overlaps with the overlapping portion in an order determined by the magnitude of the inclination of the line-of-sight direction corresponding to each of the image data; and a depth detection means for detecting the depth of the subject represented in the image data from the streak pattern represented in the epipolar plane image data. [Effects of the Invention]

[0012] According to the above aspect, it is possible to determine the position in the depth direction of a subject that appears in image data captured and generated by an existing imaging device, without adding any imaging equipment. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram illustrating an example of a configuration of an image processing system according to the present disclosure. [Figure 2] 1 is a block diagram illustrating an example of a configuration of an imaging device according to the present disclosure. [Figure 3] 10A and 10B are diagrams illustrating different imaging points depending on different line-of-sight directions of a line sensor according to the present disclosure. [Figure 4] 1 is a diagram showing the positional relationship of a plurality of image data generated by imaging by each of a plurality of line sensors according to the present disclosure; [Figure 5] 10A and 10B are diagrams illustrating differences in timing of capturing images of the same point by a line sensor according to the present disclosure. [Figure 6] 1 is a diagram (part 1) showing time-series changes in an image pickup target of a line sensor according to the present disclosure; [Figure 7] FIG. 10 is a diagram (part 2) showing time-series changes in an image pickup target of a line sensor according to the present disclosure. [Figure 8] FIG. 10 is a diagram (part 3) showing time-series changes in an image pickup target of a line sensor according to the present disclosure. [Figure 9] 10A and 10B are diagrams for explaining positional deviations that occur in a subject represented in a plurality of image data generated by imaging by each of a plurality of line sensors according to the present disclosure. [Figure 10] 1 is a block diagram illustrating an example of an internal configuration of an image processing device according to the present disclosure, and a connection relationship between the image processing device, a ground station device, and a control device. [Figure 11] 10 is a flowchart illustrating an example of a processing flow by an image processing device according to the present disclosure. [Figure 12] FIG. 10 is a diagram illustrating an example of processing by an epipolar plane image generating unit according to the present disclosure. [Figure 13] FIG. 10 is a diagram illustrating an example of epipolar plane image data generated by an epipolar plane image generating unit according to the present disclosure. [Figure 14] 10A and 10B are diagrams illustrating an example of a method for quantifying a streak pattern that appears in epipolar plane image data according to the present disclosure. [Figure 15] 1 is a block diagram illustrating an example of an internal configuration of an image processing device according to the present disclosure, and a connection relationship between the image processing device, a ground station device, and a control device. [Figure 16] FIG. 1 is a block diagram illustrating an example of a hardware configuration of an image processing device according to the present disclosure. [Figure 17] 1 is a block diagram illustrating an example of a configuration of an image processing device according to the present disclosure. [Figure 18] 10 is a flowchart illustrating an example of a processing flow by an image processing device according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0014] Each embodiment will be described below with reference to the drawings. In all drawings, the same or corresponding components are designated by the same reference numerals, and common descriptions will be omitted.

[0015] First Embodiment An embodiment according to the present disclosure will be described below with reference to the drawings. As shown in Fig. 1, an image processing system 1 includes an artificial satellite 2 (hereinafter simply referred to as the satellite 2), a control device 3, an imaging device 4, a ground station device 9, and an image processing device 10. The satellite 2 moves, for example, along an orbit around the Earth. The control device 3 and the imaging device 4 are provided on the satellite 2 and operate by receiving a supply of power from a solar cell (not shown) provided on the satellite 2. The ground station device 9 and the image processing device 10 are provided, for example, in a building of a ground station 5 of the satellite 2.

[0016] The control device 3 and the ground station device 9 transmit and receive signals and data to and from each other via wireless communication. The ground station device 9 is a device that remotely controls the satellite 2, for example, in response to a user's operation. The ground station device 9 stores various information in an internal storage area, including information indicating the orbit of the satellite 2 (hereinafter referred to as orbit information) and information regarding the imaging device 4 provided on the satellite 2. The control device 3 controls the satellite 2, for example, by receiving a control signal transmitted by the ground station device 9, or autonomously. The control device 3 transmits data such as satellite images captured and generated by the imaging device 4 to the ground station device 9.

[0017] (Configuration of imaging device) The imaging device 4 is, for example, a passive visible optical multiband sensor that captures images to generate satellite image data. The imaging device 4 includes, for example, J line sensors 20-1, 20-2, ..., 20-J (where J is an integer equal to or greater than 2) arranged in parallel so that both ends are aligned, as shown in Fig. 2, and a control unit 21.

[0018] Each of the line sensors 20-1 to 20-J is an imaging unit that captures visible light in a different wavelength band and captures an image. In order to capture images of different visible light wavelength bands, a bandpass filter that transmits light in the wavelength band of the respective imaging target is provided in front of the light receiving surface of each of the line sensors 20-1 to 20-J.

[0019] The CCD (Charge Coupled Device) elements included in the line sensors 20-1 to 20-J have the same configuration. As an example, the configuration of line sensor 20-j (where j is any integer between 1 and J), which is any one of the line sensors 20-1 to 20-J, will be described. As shown in FIG. 2, line sensor 20-j is a sensor in which M CCD elements 30-j-1, 30-j-2, ..., 30-jM (where M is an integer greater than or equal to 2) are arranged in a line. M is, for example, a value such as "2024." Each of CCD elements 30-j-1 to 30-jM is a one-pixel CCD element. Hereinafter, any one CCD element in line sensors 20-1 to 20-J will be referred to as CCD element 30-jm (where m is any integer between 1 and M).

[0020] The control unit 21 controls each of the line sensors 20-1 to 20-J to capture an image and acquires data output by each of the line sensors 20-1 to 20-J. The imaging device 4 includes, for example, a lens 22 disposed at a position where light rays incident from outside are focused on the light-receiving surface of each of the line sensors 20-1 to 20-J, i.e., the pixels of the CCD elements 30-jm, as shown in FIG. 3. That is, light rays incident from outside pass through the lens 22 and the bandpass filter of each of the line sensors 20-1 to 20-J, and are focused on the pixels of the CCD elements 30-jm of each of the line sensors 20-1 to 20-J. Each of the CCD elements 30-jm receives the light rays focused on its pixel and accumulates electric charges.

[0021] For example, when the control device 3 provides an imaging instruction signal to the imaging device 4, the control unit 21 of the imaging device 4 receives the imaging instruction signal. Upon receiving the imaging instruction signal, the control unit 21 repeatedly supplies pulse signals in parallel to each of the line sensors 20-1 to 20-J at regular intervals for a predetermined imaging time. Each time the line sensors 20-1 to 20-J receive the pulse signal, it supplies a readout pulse signal in parallel to all of its CCD elements 30-jm. Upon receiving the readout pulse signal, each of the CCD elements 30-jm reads out the charge accumulated in each pixel and outputs the amount of readout charge as a pixel value that forms an image.

[0022] Each of line sensors 20-1 to 20-J outputs pixel values ​​output by its respective CCD elements 30-jm to control unit 21 in the order in which the CCD elements 30-jm are arranged. When control unit 21 outputs the last pulse signal in a predetermined imaging time, it captures the last pixel value output by each of line sensors 20-1 to 20-J and arranges all pixel values ​​captured during the predetermined imaging time in the order in which the CCD elements 30-jm are arranged and in chronological order for each line sensor 20-1 to 20-J, thereby generating image data for each line sensor 20-1 to 20-J. When control unit 21 receives one imaging instruction signal, it compiles J pieces of image data obtained based on that imaging instruction signal to generate one piece of satellite image data. Because each piece of image data included in the satellite image data is captured in a different wavelength band, this satellite image data is satellite image data captured in multiple bands.

[0023] When giving an imaging instruction signal to the imaging device 4, the control device 3 acquires the time from, for example, a timing means such as an internal clock, and includes the acquired time in the imaging instruction signal. When the control unit 21 receives the imaging instruction signal, it acquires the time included in the imaging instruction signal as a generation time, and includes the generation time in the satellite image data.

[0024] Control unit 21 associates each of the J pieces of image data included in the satellite image data with sensor identification information that identifies the corresponding line sensors 20-1 to 20-J. Here, line sensors 20-1 to 20-J corresponding to each of the J pieces of image data refer to the line sensors 20-1 to 20-J that output pixel values ​​that form each of the J pieces of image data. The sensor identification information for each of line sensors 20-1 to 20-J associated with each of the J pieces of image data is generated in advance and pre-recorded in an internal storage area of ​​control unit 21, and control unit 21 performs the association by referring to the internal storage area.

[0025] Control unit 21 includes, in the satellite image data, information indicating the arrangement order of line sensors 20-1 to 20-J stored in an internal storage area. The information indicating the arrangement order of line sensors 20-1 to 20-J is information indicated by sensor identification information corresponding to each of line sensors 20-1 to 20-J, and is recorded in advance in an internal storage area of ​​control unit 21.

[0026] Next, differences in image data generated by control unit 21 based on pixel values ​​output by each of line sensors 20-1 to 20-J will be described with reference to Fig. 3 to Fig. 9. Here, as an example, a case where J = 3, that is, a case where imaging device 4 is equipped with line sensors 20-1, 20-2, and 20-3, will be described. As shown in Fig. 2, line sensors 20-1 to 20-3 are arranged at positions that are slightly shifted in imaging device 4. Therefore, as shown in Fig. 3, at a certain time, points on ground 100 that are imaged by lens 22 on each of line sensors 20-1 to 20-3 are different positions such as points 40-1, 40-2, and 40-3.

[0027] 3, the direction of the solid line from line sensor 20-1 to point 40-1 is the line of sight of line sensor 20-1. The direction of the dashed line from line sensor 20-2 to point 40-2 is the line of sight of line sensor 20-2. The direction of the dashed line from line sensor 20-3 to point 40-3 is the line of sight of line sensor 20-3.

[0028] These line of sight directions are represented by, for example, the angle formed between the direction of the optical axis of the optical system of the imaging device 4 and each line of sight direction. For example, when the direction of the optical axis of the optical system of the imaging device 4 coincides with the line of sight direction of line sensor 20-2, α2, which is the angle of the line of sight direction of line sensor 20-2, is 0°. α1, which is the angle of the line of sight direction of line sensor 20-1, is the angle formed between the line of sight direction of line sensor 20-2 and the line of sight direction of line sensor 20-1, with a minus sign added. α3, which is the angle of the line of sight direction of line sensor 20-3, is the angle formed between the line of sight direction of line sensor 20-2 and the line of sight direction of line sensor 20-3, with a plus sign added.

[0029] 3, if control device 3 issues an imaging instruction signal to imaging device 4, the imaging start point for line sensor 20-1 will be point 40-1. The imaging start point for line sensor 20-2 will be point 40-2. The imaging start point for line sensor 20-3 will be point 40-3. As described above, satellite 2 moves along an orbit around the Earth, and this movement direction is assumed to be the direction of arrow ST in FIG. 3. While line sensors 20-1 to 20-3 repeatedly take images in accordance with the imaging instruction signal, the position of satellite 2 moves in the direction of arrow ST.

[0030] Here, when the three image data 41-1, 41-2, and 41-3 generated by the control unit 21 based on the pixel values ​​output by each of the line sensors 20-1 to 20-3 are overlapped so that the latitude and longitude match, the positional relationship shown in FIG. 4 is obtained. Note that the direction of the arrow ST is assumed to be perpendicular to the longitudinal direction of each of the line sensors 20-1 to 20-J shown in FIG. 2. In this case, as shown in FIG. 4, the shape of the image data 41-1, 41-2, and 41-3 is rectangular. In FIG. 4, the solid line indicates the range of the image data 41-1 generated by imaging with the line sensor 20-1. The dashed line indicates the range of the image data 41-2 generated by imaging with the line sensor 20-2. The dashed line indicates the range of the image data 41-3 generated by imaging with the line sensor 20-3.

[0031] As shown in Fig. 4, a positional deviation occurs in the longitudinal direction of image data 41-1 to 41-3, which corresponds to the interval between points 40-1, 40-2, and 40-3. This is caused by differences in the arrangement positions of line sensors 20-1 to 20-3 on the focal plane of lens 22, in other words, differences in the line-of-sight directions of line sensors 20-1 to 20-3. Note that, for ease of viewing, Fig. 4 shows image data 41-1 to 41-3 slightly shifted in the lateral direction, but since line sensors 20-1 to 20-3 are arranged in parallel so that their ends are aligned as shown in Fig. 2, no positional deviation occurs in the lateral direction.

[0032] In FIG. 4, line image data 41s-1 including point 40-1 is generated by M pixel values ​​that are first output by line sensor 20-1 in a predetermined imaging time. Line image data 41e-1 is generated by M pixel values ​​that are last output by line sensor 20-1 in a predetermined imaging time. Line image data 41s-2 including point 40-2 is generated by M pixel values ​​that are first output by line sensor 20-2 in a predetermined imaging time. Line image data 41e-2 is generated by M pixel values ​​that are last output by line sensor 20-2 in a predetermined imaging time. Line image data 41s-3 including point 40-3 is generated by M pixel values ​​that are first output by line sensor 20-3 in a predetermined imaging time. Line image data 41e-3 is generated by M pixel values ​​that are last output by line sensor 20-3 in a predetermined imaging time.

[0033] Although not shown in Figure 4, there are multiple line image data generated during a specified imaging time between line image data 41s-1 and line image data 41e-1, between line image data 41s-2 and line image data 41e-2, and between line image data 41s-3 and line image data 41e-3, and the number of line image data is the same for image data 41-1, 41-2, and 41-3.

[0034] Next, as shown in FIG. 5, when a building 90 exists on the ground 100, differences in the timing at which line sensors 20-1 to 20-3 capture images of the building 90 will be described. Satellites 2-t1, 2-t2, and 2-t3 indicate the position of satellite 2 at times t1, t2, and t3, respectively. Time t3 is later than time t2, which is later than time t1. At time t1, an image of building 90 is formed on line sensor 20-3. At time t2, an image of building 90 is formed on line sensor 20-2. At time t3, an image of building 90 is formed on line sensor 20-1. In this way, the timing at which the same object, building 90, is captured by line sensors 20-1 to 20-3 is different. This results in phenomena such as those shown in FIGS. 6 to 8.

[0035] 6 to 8 show a state in which four buildings 91, 92, 93, and 94 exist on the ground 100, and a cloud 80 is floating in the air. In FIG. 6, a building 92 is installed in place of building 90 in the position of building 90 in FIG. 5. In this case, as shown in FIG. 6, at time t1, an image of building 92 is formed on line sensor 20-3. At time t2, an image of building 92 is formed on line sensor 20-2. In contrast, at time t3, line sensor 20-1 is positioned at a position where an image of building 92 is formed, but the reflected light from building 92 is blocked by cloud 80. Therefore, an image of building 92 is not formed on line sensor 20-1, and an image of cloud 80 is formed on line sensor 20-1.

[0036] Next, in Figure 7, satellites 2-t4, 2-t5, and 2-t6 indicate the position of satellite 2 at times t4, t5, and t6, respectively. Time t6 is later than time t5, which is later than time t4. At time t4, an image of building 93 is formed on line sensor 20-3. In contrast, at times t5 and t6, line sensors 20-2 and 20-1 are positioned to form an image of building 93, but cloud 80 blocks the reflected light from building 93. Therefore, an image of building 93 is not formed on line sensors 20-2 and 20-1, but an image of cloud 80 is formed instead.

[0037] Next, in Figure 8, satellites 2-t7, 2-t8, and 2-t9 indicate the position of satellite 2 at times t7, t8, and t9, respectively. Time t9 is later than time t8, which is later than time t7. At times t7, t8, and t9, line sensors 20-3, 20-2, and 20-1 are positioned such that an image of building 94 is formed, but cloud 80 blocks the reflected light from building 94. Therefore, line sensors 20-3, 20-2, and 20-1 do not form an image of building 94, but form an image of cloud 80 instead.

[0038] The buildings 91 to 94 and cloud 80 that appear in image data 41-1, 41-2, and 41-3 corresponding to line sensors 20-1 to 20-3, respectively, captured at the timings shown in Figures 6 to 8, have a positional relationship as shown in Figure 9. Note that, although dotted lines indicating the timings of times t1 to t9 are shown in image data 41-1 to 41-3 in Figure 9, these dotted lines are not included in image data 41-1 to 41-3.

[0039] As shown in Fig. 9(a), building 91 appears in image data 41-1 corresponding to line sensor 20-1, but buildings 92 to 94 are hidden by clouds 80 as shown by dashed lines. As shown in Fig. 9(b), buildings 91 and 92 appear in image data 41-2 corresponding to line sensor 20-2, but buildings 93 and 94 are hidden by clouds 80 as shown by dashed lines. As shown in Fig. 9(c), buildings 91, 92, and 93 appear in image data 41-3 corresponding to line sensor 20-3, but building 94 is hidden by cloud 80 as shown by dashed lines.

[0040] 9(a), (b), and (c) are overlaid so that the latitudes and longitudes match, as shown in FIG. 4. However, the positions of the buildings 91 to 94 shown in the image data 41-1, 41-2, and 41-3 match, but a positional shift occurs for the cloud 80. That is, an absolute positional shift occurs in the image data 41-1, 41-2, and 41-3, for example, in the position of the building 91 due to parallax, which is the difference in the line-of-sight directions of the line sensors 20-1 to 20-3. In addition to this absolute positional shift, a relative positional shift occurs due to the parallax, in which, when the positions of the buildings 91 to 94 are used as a reference, the position of the cloud 80, which is located at an altitude where the distance to the satellite 2 is shorter than that of the buildings 91 to 94, is shifted. Therefore, by measuring the relative positional shift of the cloud 80, it is possible to estimate the depth position of the subject cloud 80, i.e., the altitude of the cloud 80, based on the same principle as stereoscopic vision using a stereo pair of images.

[0041] 6 to 8, the position of the cloud 80 is shown as being stationary at all times t1 to t9, but in reality, the position of the cloud 80 fluctuates as the time changes. However, since the moving speed of the satellite 2 is much faster than the moving speed of the cloud 80, here, the cloud 80 is considered to be stationary during the time when the imaging device 4 is imaging in response to one imaging command signal.

[0042] (Configuration of image processing device) 10 is a diagram showing the internal configuration of the image processing device 10, the connection between the image processing device 10 and the ground station device 9, and the connection between the ground station device 9 and the control device 3. The image processing device 10 includes an image acquisition unit 11, an information acquisition unit 12, an image superimposition unit 13, an epipolar plane image generation unit 14, and a depth detection unit 15.

[0043] The image acquisition unit 11 is connected to the ground station device 9 and acquires satellite image data from the ground station device 9. From the acquired satellite image data, the image acquisition unit 11 acquires a combination of J pieces of image data and sensor identification information, information indicating the arrangement order of the line sensors 20-1 to 20-J, and the generation time.

[0044] The information acquisition unit 12 is connected to the ground station device 9 and acquires from the ground station device 9 orbit information of the satellite 2 stored in an internal storage area of ​​the ground station device 9 and information indicating the line of sight direction of each of the line sensors 20-1 to 20-J. Note that each piece of information indicating the line of sight direction of each of the line sensors 20-1 to 20-J is pre-associated with corresponding sensor identification information. The orbit information of the satellite 2 includes information indicating the position and velocity of the satellite 2 in chronological order. Note that the chronological order is shown in association with the time. The time of the clock means provided inside the control device 3 of the satellite 2 is pre-set so that by referring to the orbit information based on the time of the clock means provided inside the control device 3, the accurate position and velocity of the satellite 2 at that time can be obtained. As described above, the information indicating the line of sight direction of each of the line sensors 20-1 to 20-J is information indicating the line of sight direction in degrees.

[0045] The image superimposing unit 13 superimposes the J pieces of image data acquired by the image acquiring unit 11 by aligning the pieces of image data so as to eliminate any absolute positional deviation that occurs due to different line-of-sight directions of the line sensors 20-1 to 20-J. To perform this alignment, the image superimposing unit 13 projects each of the J pieces of image data onto a predetermined geodetic system, for example. Here, the predetermined geodetic system is, for example, the WGS (World Geodetic System) 84 ellipsoid. By projecting the pieces of image data onto the predetermined geodetic system, the latitude and longitude of each pixel are identified, and the identified latitude and longitude are associated with each pixel of the J pieces of image data. The image superimposing unit 13 superimposes the J pieces of image data by matching the latitude and longitude of the pieces of image data.

[0046] The epipolar planar image generation unit 14 detects overlapping portions of the J pieces of image data superimposed by the image superimposition unit 13. The epipolar planar image generation unit 14 selects a transverse line that crosses the detected overlapping portion along the direction in which the ranges of the J pieces of image data in the superimposed state shift. The direction in which the ranges of the J pieces of image data in the superimposed state shift is the direction in which the satellite 2 moves. The epipolar planar image generation unit 14 extracts image data of the portion where the selected transverse line overlaps with the overlapping portion.

[0047] The epipolar plane image generating unit 14 generates epipolar plane image data by arranging each piece of image data of the overlapping portion of the transverse line in an order determined by the magnitude of the inclination of the line-of-sight direction of the line sensors 20-1 to 20-J that generated each piece. The order determined by the magnitude of the inclination of the line-of-sight direction of the line sensors 20-1 to 20-J is the arrangement order of the line sensors 20-1 to 20-J. For example, in the example described with reference to FIG. 3, the line-of-sight direction angle α1 of the line sensor 20-1 is a negative value, the line-of-sight direction angle α2 of the line sensor 20-2 is 0°, and the line-of-sight direction angle α3 of the line sensor 20-3 is a positive value. In this case, arranging the line sensors 20-1, 20-2, and 20-3 in this order means arranging them in order from the line sensor with the smallest inclination of the line-of-sight direction.

[0048] The depth detection unit 15 includes a quantification unit 16 and a depth calculation unit 17, and detects the depth of the subject appearing in the image data from the streak pattern appearing in the epipolar plane image data. The quantification unit 16 quantifies the lines forming the streak pattern based on information indicating the line of sight direction of each of the line sensors 20-1 to 20-J acquired by the information acquisition unit 12 and the lines forming the streak pattern appearing in the epipolar plane image data generated by the epipolar plane image generation unit 14.

[0049] The depth calculation unit 17 acquires information indicating the altitude of the satellite 2, which is determined from the orbit information acquired by the information acquisition unit 12 and the generation time acquired by the image acquisition unit 11. The depth calculation unit 17 calculates the absolute depth, i.e., the altitude, of subjects such as buildings 90 and clouds 80 that appear in the image data, based on the acquired information indicating the altitude and the value obtained by quantification by the quantification unit 16. The depth calculation unit 17 outputs the calculated depth distribution as a depth distribution.

[0050] <Processing of First Embodiment> The processing by the image processing device 10 will be described below with reference to FIGS. 11 to 13. In the following description, the pixel value output by the line sensor 20-j is represented by I(j;m,n). In I(j;m,n), "j" is the sensor identification information described above, and "j" is an integer between 1 and J. "m" is a pixel number, i.e., a number identifying a pixel, and corresponds to "m" used in the code of the CCD element 30-jm. As described above, "m" is an integer between 1 and M. "n" is a line number, and "n" is an integer between 0 and (N-1). "N" is the number of line image data included in the image data 41-j generated by capturing an image with the line sensor 20-j. For example, in the image data 41-1 shown in FIG. 4, n=0 corresponds to the line image data 41s-1, and n=N-1 corresponds to the line image data 41e-1. Therefore, when "j", "m", and "n" in I(j;m,n) are determined, the pixel value of one pixel included in the satellite image data is identified.

[0051] When the control unit 21 of the imaging device 4 receives an imaging instruction signal from the control device 3 and generates satellite image data as described above, the control unit 21 outputs the generated satellite image data to the control device 3. The control device 3 imports the satellite image data output by the control unit 21 and transmits the imported satellite image data to the ground station device 9. The ground station device 9 receives the satellite image data transmitted by the control device 3 and records the received satellite image data in an internal storage area. In this state, processing by the image processing device 10 shown in FIG. 11 begins.

[0052] For example, in response to an operation by a user of the image processing device 10, the image acquisition unit 11 outputs a signal requesting satellite image data to the ground station device 9. When the ground station device 9 receives the signal from the image acquisition unit 11, it reads the satellite image data from its internal storage area and outputs it to the image acquisition unit 11. When the image acquisition unit 11 imports the satellite image data output by the ground station device 9, it acquires from the imported satellite image data a combination of J pieces of image data and sensor identification information, information indicating the arrangement order of the line sensors 20-1 to 20-J, and the generation time. The image acquisition unit 11 outputs the combination of J pieces of image data and sensor identification information, and the information indicating the arrangement order of the line sensors 20-1 to 20-J to the image superimposition unit 13. The image acquisition unit 11 outputs the generation time to the information acquisition unit 12 (S1).

[0053] When the information acquisition unit 12 receives the generation time output by the image acquisition unit 11, it outputs a signal to the ground station device 9 requesting orbit information of the satellite 2 and information indicating the line-of-sight direction of each of the line sensors 20-1 to 20-J. When the ground station device 9 receives this signal from the information acquisition unit 12, it reads the orbit information of the satellite 2 and information indicating the line-of-sight direction of each of the line sensors 20-1 to 20-J from an internal storage area and outputs the information to the information acquisition unit 12. The information acquisition unit 12 receives the orbit information of the satellite 2 and information indicating the line-of-sight direction of each of the line sensors 20-1 to 20-J output by the ground station device 9. From the received orbit information, the information acquisition unit 12 identifies the position and velocity of the satellite 2 at the received generation time and at times before and after the generation time. Based on the identified position and velocity of the satellite 2, the information acquisition unit 12 calculates the movement direction of the satellite 2 and the altitude of the satellite 2 at the generation time. The information acquisition unit 12 outputs information indicating the calculated moving direction of the satellite 2 to the epipolar plane image generation unit 14. The information acquisition unit 12 outputs the acquired information indicating the line of sight direction of each of the line sensors 20-1 to 20-J to the quantification unit 16. The information acquisition unit 12 outputs information indicating the calculated altitude of the satellite 2 to the depth calculation unit 17 (S2-1).

[0054] The image superimposing unit 13 receives a combination of the J pieces of image data and sensor identification information output by the image acquiring unit 11, and information indicating the arrangement order of the line sensors 20-1 to 20-J. The image superimposing unit 13 projects each of the J pieces of image data onto a predetermined geodetic system and superimposes the J pieces of image data by matching the latitude and longitude of the pieces of image data. As a result, for example, when J=3, the image data 41-1, 41-2, and 41-3 shown in FIG. 9 are superimposed such that the position of the building 91 matches in the image data 41-1, 41-2, and 41-3, and the positions of the buildings 91 and 92 match in the image data 41-2 and 41-3. Note that the superimposition by the image superimposing unit 13 may be performed in an order that follows the arrangement order of the line sensors 20-1 to 20-J, or may be performed in an order that does not follow the arrangement order. The image superimposing unit 13 outputs the superimposed image data (hereinafter referred to as superimposed image data), sensor identification information associated with each of the image data, and information indicating the arrangement order of the line sensors 20-1 to 20-J to the epipolar plane image generating unit 14 (S2-2).

[0055] The epipolar planar image generation unit 14 receives the superimposed image data output by the image superimposition unit 13, the sensor identification information associated with each piece of image data included in the superimposed image data, and information indicating the arrangement order of the line sensors 20-1 to 20-J. The epipolar planar image generation unit 14 receives the information indicating the moving direction of the satellite 2 output by the information acquisition unit 12 in the process of S2-1. The epipolar planar image generation unit 14 detects overlapping portions, which are portions common to all of the image data 41-1 to 41-J, in the received superimposed image data. For example, when J=3, it is assumed that the image data 41-1 to 41-3 forming the superimposed image data are overlapped in the arrangement shown in FIG. 12. In this case, the epipolar planar image generation unit 14 detects the hatched portion indicated by the reference numeral 50 as an overlapping portion (S3).

[0056] The epipolar plane image generating unit 14 determines an imaging point in the superimposed image data. Here, the imaging point is a position in the image data 41-j that corresponds to the center point of the pixel of the J×M CCD elements 30-jm provided in the line sensors 20-1 to 20-J. The imaging point is specified by the latitude and longitude obtained by projecting the position of the center point of the pixel of the CCD element 30-jm onto a predetermined geodetic system. In the superimposed image data, there are J×M×N imaging points, the number of which is the number of combinations of (j;m,n). Hereinafter, the coordinates of the imaging point will be referred to as (latitude j;m,n ,longitude j;m,n The epipolar plane image generating unit 14 detects the direction indicated by the captured information indicating the moving direction of the satellite 2 and the trajectory of the imaging point moving in the opposite direction (hereinafter, this trajectory will be referred to as the imaging trajectory).

[0057] For example, the epipolar plane image generating unit 14 determines the point indicated by the reference numeral 60 in FIG. 12 as the imaging point (hereinafter referred to as imaging point 60) in the image data 41-3, and the coordinates of this imaging point 60 are expressed using j=3, m=1000, and n=10 (latitude 3;1000,10 ,longitude 3;1000,10 In this case, the epipolar plane image generating unit moves the coordinates of the imaging point to a position that exists in the direction indicated by the captured information indicating the moving direction of satellite 2 and in the opposite direction to that direction, and that is included in each of the N line image data that form image data 41-3.

[0058] The image data 41-3 is image data obtained by moving the line sensor 20-3 in the direction of movement of the satellite 2. Therefore, when the coordinates of the imaging point are moved in the direction of movement of the satellite 2, the value of "n", which is the line number, changes, but the value of "m", which is the pixel number, does not change. Therefore, the set of coordinates of N points including the coordinates of the imaging point is expressed as {(latitude 3;1000,n ,longitude 3;1000,n ) where n = 0, ..., N-1}. This set of coordinates {(latitude 3;1000,n ,longitude 3;1000,n ) where n=0, . . . , N−1}, an imaging trace 61 corresponding to an imaging point 60 is specified.

[0059] The epipolar planar image generation unit 14 detects a set of coordinates that identify each of the J×M imaging tracks identified from each of the J×M×N imaging points. The epipolar planar image generation unit 14 selects imaging tracks that are common to the image data 41-1 to 41-J from the detected J×M imaging tracks. Here, imaging tracks that are common to the image data 41-1 to 41-J refer to imaging tracks in which the coordinates of the range of the overlapping portion of each of the imaging tracks of the image data 41-1 to 41-J all match. Therefore, imaging tracks that are common to the image data 41-1 to 41-J are imaging tracks that cross the overlapping portion in the direction along the movement direction of the satellite 2 (S4).

[0060] The epipolar plane image generating unit 14 selects one of the selected imaging traces that is not being processed as a transverse line (S5). The epipolar plane image generating unit 14 extracts image data of the portion where the transverse line and the overlapping portion overlap. For example, in the example shown in FIG. 12, it is assumed that the epipolar plane image generating unit 14 selects the imaging trace 61 as the transverse line. In this case, the portion where the imaging trace 61 and the overlapping portion 50 overlap is the portion indicated by the reference numeral 70 of the imaging trace 61 (hereinafter, this portion is referred to as the overlapping line 70). The overlapping line 70 can be expressed as a set of coordinates as follows: {(latitude 3;1000,n ,longitude 3;1000,n ) where n=0, ..., Ne}, where "Ne" is the line number of the line image data of the image data 41-3 that matches the position of the last line image data 41e-1 of the image data 41-1 in FIG.

[0061] The epipolar plane image generating unit 14 extracts image data of the portion where the crossing line and the overlapping portion overlap. In the example shown in FIG. 12, the epipolar plane image generating unit 14 extracts image data of the overlapping line 70. As described above, the overlapping line 70 is a set of coordinates (latitude, 3;1000,n ,longitude 3;1000,n) [where n = 1, ..., Ne], and coordinates included in this set of coordinates exist in all of the image data 41-1, 41-2, and 41-3. Therefore, the epipolar planar image generation unit 14 detects pixel values ​​for each of the coordinates of the overlapping line 70 from each of the image data 41-1, 41-2, and 41-3. The epipolar planar image generation unit 14 arranges the detected pixel values ​​for each of the image data 41-1, 41-2, and 41-3 in the order of the coordinates, and generates a combination including multiple pixel values. Hereinafter, this combination including multiple pixel values ​​is referred to as extracted line image data. The epipolar planar image generation unit 14 associates the generated extracted line image data with corresponding sensor identification information (S6).

[0062] Based on the information indicating the arrangement order of the line sensors 20-1 to 20-J acquired in the process at S3 and the sensor identification information associated with each piece of extracted line image data, the epipolar plane image generation unit 14 arranges the extracted line image data in that arrangement order to generate epipolar plane image data. Arranging the extracted line image data in the arrangement order of the line sensors 20-1 to 20-J means, as described above, arranging the extracted line image data in an order determined by the magnitude of the inclination of the line-of-sight direction of the line sensors 20-1 to 20-J. By arranging the extracted line image data in this order, a stripe pattern with a different inclination for each subject appears in the epipolar plane image data, depending on the distance from the imaging position to the subject.

[0063] FIG. 13 is a diagram showing an example of epipolar generated image data generated when an imaging trail passing through all of the buildings 91, 92, 93, and 94 and cloud 80 shown in FIGS. 9(a), (b), and (c) is selected as the crossing line. In FIG. 13, the solid line indicated by reference numeral 70-1 shows colors corresponding to the pixel values ​​included in the extracted line image data extracted from image data 41-1. The dashed line indicated by reference numeral 70-2 shows colors corresponding to the pixel values ​​included in the extracted line image data extracted from image data 41-2. The dashed-dotted line indicated by reference numeral 70-3 shows colors corresponding to the pixel values ​​included in the extracted line image data extracted from image data 41-3. Note that when the pixel values ​​are integer values ​​between 0 and 255, for example, the colors corresponding to the pixel values ​​are any of the 256 grayscale colors.

[0064] As shown in Fig. 9(a), a building 91 and clouds 80 are depicted in image data 41-1. Therefore, on the line indicated by reference numeral 70-1 in Fig. 13, the portion indicated by reference numeral 91s-1 corresponding to the part of the building 91 is shown in a grayscale color corresponding to the color of the building 91. The portion indicated by reference numeral 80s-1 corresponding to the part of the clouds 80 is shown in a grayscale color corresponding to the color of the clouds 80. The parts of reference numeral 70-1 other than reference numerals 91s-1 and 80s-1 are shown in a grayscale color corresponding to, for example, the color of the ground 100.

[0065] As shown in FIG. 9(b), buildings 91 and 92 and cloud 80 are depicted in image data 41-2. Therefore, on the line indicated by reference numeral 70-2 in FIG. 13, the portion indicated by reference numeral 91s-2 corresponding to the portion of building 91 is shown in a grayscale color corresponding to the color of building 91. The portion indicated by reference numeral 92s-2 corresponding to the portion of building 92 is shown in a grayscale color corresponding to the color of building 92. The portion indicated by reference numeral 80s-2 corresponding to the portion of cloud 80 is shown in a grayscale color corresponding to the color of cloud 80. The portions of reference numeral 70-2 other than reference numerals 91s-2, 92s-2, and 80s-2 are shown in a grayscale color corresponding to, for example, the color of the ground surface of the ground 100.

[0066] As shown in FIG. 9(c), buildings 91, 92, and 93 and cloud 80 are depicted in image data 41-3. Therefore, on the line indicated by reference numeral 70-3 in FIG. 13, the portion indicated by reference numeral 91s-3 corresponding to the portion of building 91 is shown in a grayscale color corresponding to the color of building 91. The portion indicated by reference numeral 92s-3 corresponding to the portion of building 92 is shown in a grayscale color corresponding to the color of building 92. The portion indicated by reference numeral 93s-3 corresponding to the portion of building 93 is shown in a grayscale color corresponding to the color of building 93. The portion indicated by reference numeral 80s-3 corresponding to the portion of cloud 80 is shown in a grayscale color corresponding to the color of cloud 80. The portions of reference numeral 70-3 other than reference numerals 91s-3, 92s-3, 93s-3, and 80s-3 are shown in a grayscale color corresponding to, for example, the color of the ground surface of the ground 100.

[0067] Note that in FIG. 13, for ease of viewing, the lines indicated by the reference numerals 70-1, 70-2, and 70-3 are shown spaced apart. However, in actual epipolar plane image data, the lines indicated by the reference numerals 70-1, 70-2, and 70-3 are adjacent to each other with no gaps between them. Therefore, stripes appear at the locations indicated by the reference numerals 111 and 112 at both ends of the portions indicated by the reference numerals 80s-1, 80s-2, and 80s-3 corresponding to the cloud 80. Similarly, stripes appear at the locations indicated by the reference numerals 113 and 114 at both ends of the portions indicated by the reference numerals 92s-2 and 92s-3 corresponding to the building 92. Stripes appear at the locations indicated by the reference numerals 115 and 116 at both ends of the portions indicated by the reference numerals 91s-1, 91s-2, and 91s-3 corresponding to the building 91. In reality, J is not a small number such as 3, but a large value, so the vertical length of the epipolar plane image data is longer than in the example shown in Fig. 13, and the lines forming the stripe pattern appear more clearly. In this case, if the building 93 is expressed in other extracted line image data, a portion similar to the portion of reference symbol 93s-3 will exist, and stripe patterns will also appear on both ends of reference symbol 93s-3.

[0068] Here, let us say that each of the J extracted line image data forming the epipolar plane image data is represented by EPI(j,p). The extracted line image data EPI(j,p) is defined by the following equation (1) using I(j;m,n), which is the pixel value of the image data 41-1 to 41-J.

[0069] EPI(j,p) = I(j;m q ,n j )…(1)

[0070] On the right side of equation (1), "m q " is the pixel number of the pixel selected as the imaging point, and is a fixed value in J pieces of extracted line image data EPI(j,p) forming one piece of epipolar plane image data. j " is Ns j ~Ne j is an integer between "Ns j " is the line number of the start point of the overlapping line in the image data 41-j corresponding to the line sensor 20-j. j " is the line number of the end point of the overlapping line in the image data 41-j corresponding to the line sensor 20-j. In the left side of equation (1), j=1 to J, and p=n j -Ns j and "p" is 0~(Ne j -Ns j ) where "Ne j -Ns j " + "1" corresponds to the number of pixels contained in the overlapping line, so for all J, "Ne j -Ns j +1" will be the same value. j -Ns j "+1" is represented as "Nt".

[0071] The epipolar plane image generating unit 14 generates the epipolar plane image data, that is, J pieces of extracted line image data EPI(j,p), and p " and "Ns" for all J j " is output to the quantification unit 16 (S7).

[0072] The quantification unit 16 compares the epipolar plane image data output by the epipolar plane image generation unit 14 with the "m p " and "Ns" for all J j The quantification unit 16 extracts lines forming the streak pattern expressed in the epipolar plane image data by a predetermined image processing (S8). Here, the predetermined image processing is, for example, the Hough transform method.

[0073] The quantification unit 16 acquires information indicating the line of sight direction of each of the line sensors 20-1 to 20-J output by the information acquisition unit 12 in the process of S2-1. The quantification unit 16 quantifies each of the lines forming the extracted streak pattern using information indicating the line of sight direction of each of the line sensors 20-1 to 20-J. Incidentally, in FIG. 13, for the sake of convenience of explanation, the lines of the streak pattern indicated by the reference numerals 111, 112, 113, 114, 115, and 116 are shown as straight lines, but these lines of the streak pattern will not become straight lines if the extracted line image data are simply lined up. In order to make the lines of the streak pattern appear straight lines, the extracted line image data is quantified by adjusting the line of sight direction angle α as shown in FIG. j It is necessary to project the extracted line image data EPI(j,p) onto a plane with the horizontal axis at α and the vertical axis at p. In FIG. 14, the minimum value of the extracted line image data EPI(j,p) in the direction of the p axis is "0" and the maximum value is "Nt-1". This makes it possible to detect the stripe pattern lines 111, 112, 113, 114, 115, and 116 as straight lines. Therefore, in the predetermined image processing performed by the quantification unit 16, the extracted line image data is projected onto a plane with the line of sight direction angle α j The image processing is performed to extract the straight lines of the stripe pattern by projecting the image data onto a plane with the horizontal axis and the vertical axis. j By projecting onto a plane with the horizontal axis and p as the vertical axis, the lines of the stripe pattern appear as straight lines when the value of p of the same subject and the line of sight angle α j This is because there is a linear relationship between the value of

[0074] The quantification unit 16 calculates the line-of-sight angle α α , which is included in the information indicating the line-of-sight direction of each of the line sensors 20-1 to 20-J, for each of the stripe-patterned lines 111, 112, 113, 114, 115, and 116 extracted by predetermined image processing. j Either of these will result in p=Aα j +B. As shown in FIG. 14, the line of sight angle α j In a plane with the horizontal axis being a and the vertical axis being p, coefficient A is a coefficient that indicates the slope, and coefficient B is a coefficient that indicates the intercept.

[0075] The coefficients A and B detected in this way, and α j , and the α j The value of the subscript j in the equation (1) is a value obtained by quantifying each of the lines 111, 112, 113, 114, 115, and 116 that form the stripe pattern. j All combinations of , and j, and "m p " and "Ns" for all J j " is output to the depth calculation unit 17 (S9).

[0076] The depth calculation unit 17 calculates the coefficients A, B, and α output by the quantification unit 16. j All combinations of , and j, and "m p " and "Ns" for all J j The depth calculation unit 17 takes in the information indicating the altitude of the satellite 2 output by the information acquisition unit 12 in the process of S2-1. For each of the taken-in combinations, the depth calculation unit 17 calculates the coefficient A and the α corresponding to the coefficient A. j and the information indicating the captured altitude, the absolute depth for each combination, that is, the distance from the satellite 2 to the subject, is calculated.

[0077] The depth calculated by the depth calculation unit 17 is calculated using the coefficient A and α j and "m p " and "α j "Ns" corresponding to the value of "j" included in the combinationj " and "α j " and the coefficient B included in the combination (j;m p ,p+Ns j ) [where p=Aα j +B]. Therefore, the depth calculation unit 17 calculates the depth as a depth distribution DepthMap(j;m p ,Aα j +B+Ns j ) and record it as a value (S10).

[0078] The depth calculation unit 17 outputs a processing continuation instruction signal to the epipolar plane image generation unit 14. Upon receiving the processing continuation instruction signal from the depth calculation unit 17, the epipolar plane image generation unit 14 determines whether all of the imaging traces selected in the processing of S4 have been selected as transverse lines (S11). If the epipolar plane image generation unit 14 determines that all of the imaging traces have not been selected as transverse lines (S11, No), it performs the processing of S5 again. On the other hand, if the epipolar plane image generation unit 14 determines that all of the imaging traces have been selected as transverse lines (S11, Yes), it outputs a processing end notification signal to the depth calculation unit 17.

[0079] When the depth calculation unit 17 receives a processing completion notification signal from the epipolar plane image generation unit 14, it calculates the depth distribution DepthMap(j;m p ,Aα j +B+Ns j ) is output (S12), and the process ends.

[0080] <Effects of the First Embodiment> The output depth map (j;m p ,Aα j +B+Ns j), it is possible to grasp the absolute depth of the subject shown in each of the image data 41-1 to 41-J generated by imaging by each of the line sensors 20-1 to 20-J, i.e., the accurate distance from the satellite 2 to the subject. In other words, by using the image processing device 10, it is possible to determine the position in the depth direction of the subject shown in the image data 41-1 to 41-J captured and generated by the existing imaging device 4, without adding any imaging equipment to the satellite 2. By referring to this depth, for example, in the example shown in FIG. 9, it is possible to grasp that the depth of the cloud 80 is smaller than the depth of the buildings 91 to 93 installed on the ground 100, and therefore that the cloud 80 is located at an altitude higher than the ground 100. Therefore, it is possible to grasp the depth distribution DepthMap(j;m p ,Aα j +B+Ns j ), by setting an appropriate threshold value and dividing the area, it is possible to segment, for example, the ice and snow part on the ground 100 and the cloud part in the air.

[0081] Second Embodiment An embodiment of the present disclosure will be described with reference to the drawings. An example will be described in which an image processing device 10a shown in FIG. 15 is used instead of the image processing device 10 shown in FIG. 10. Hereinafter, for convenience of explanation, an image processing system in which the image processing device 10 has been replaced with the image processing device 10a will be referred to as the image processing system 1a. The image processing system 1a includes a satellite 2, a control device 3, an imaging device 4, a ground station device 9, and the image processing device 10a. The image processing device 10a includes an image acquisition unit 11, an image superposition unit 13, an epipolar plane image generation unit 14a, and a depth detection unit 15a. That is, the image processing device 10a does not include the information acquisition unit 12 included in the image processing device 10 shown in FIG. 10. Therefore, the image processing device 10a generates a depth distribution of satellite image data without using the orbit information of the satellite 2 acquired by the information acquisition unit 12, information indicating the line-of-sight direction of each of the line sensors 20-1 to 20-J, and the movement direction and altitude of the satellite 2 calculated by the information acquisition unit 12.

[0082] The epipolar planar image generating unit 14a has the same configuration as the epipolar planar image generating unit 14, except for the configuration described below. Specifically, in the process of S4 in FIG. 11, the epipolar planar image generating unit 14 selects common imaging traces in the image data 41-1 to 41-J, and in the process of S5, selects one imaging trace that is not the target of processing from the selected imaging traces as a crossing line. In contrast, the epipolar planar image generating unit 14a selects, as the crossing line, a line that crosses the overlapping portion detected in the process of S3 and that runs along the direction in which the ranges of the image data 41-1 to 41-J in the superimposed state shift. The direction in which the ranges of the image data 41-1 to 41-J in the superimposed state shift can be detected, for example, as the direction of a line segment connecting the positions of pixels in the image data 41-1 to 41-J that have the same pixel number and line number.

[0083] The depth detection unit 15a includes a quantification unit 16a and a depth recording unit 18. The quantification unit 16a has the same configuration as the quantification unit 16 except for the configuration described below. The quantification unit 16 acquires the gaze direction angle α j In contrast, the quantification unit 16a quantifies the lines forming the stripe pattern using the line of sight direction angle α j Instead, multiple gaze direction angles α j is appropriately determined, and the coefficients A and B are detected in the same manner as in the first embodiment.

[0084] The depth recording unit 18 receives the coefficients A, B, and α output from the quantification unit 16a. j All combinations of , and j, and "m p " and "Ns" for all J j The depth recording unit 18 records each of the coefficients A as a coefficient B corresponding to the coefficient A and an α j and "m p " and "Ns" corresponding to the value of "j" corresponding to the coefficient A j " and (j;m p ,p+Ns j) [where p=Aα j +B], and the coefficient A is recorded in the corresponding location of the depth distribution DepthMap.

[0085] <Processing of the second embodiment> The processing by the image processing device 10a is the same as the processing by the image processing device 10 shown in Fig. 11 except for the processing of S2-1 and S4, in which the epipolar plane image generator 14a selects a transverse line in the above-described procedure in the processing of S5. The processing of S6 and S7 is performed by replacing the epipolar plane image generator 14 with the epipolar plane image generator 14a. The processing of S8 is performed by replacing the quantifier 16 with the quantifier 16a.

[0086] In the process of S9, the quantification unit 16a calculates the coefficients A, B, and α in the above-described procedure. j , and j are detected as values ​​obtained by quantifying them, and the detected coefficients A, B, α j、 All combinations of and j and "m p " and "Ns" for all J j " is output to the depth recording unit 18.

[0087] In the process of S10, the depth recording unit 18 calculates the coefficients A, B, and α output by the quantification unit 16a. j、 All combinations of and j and "m p " and "Ns" for all J j The depth recording unit 18 takes in the coefficient A as a value indicating the relative depth, and calculates the coefficient A as a value indicating the depth distribution DepthMap(j;m p ,p+Ns j ) [where p=Aα j+B]. In the process of S11, the depth recording unit 18 outputs a processing continuation instruction signal to the epipolar planar image generating unit 14a. When the epipolar planar image generating unit 14a receives the processing continuation instruction signal from the depth recording unit 18, it determines whether or not there is a transverse line that can be selected using the procedure described above. If the epipolar planar image generating unit 14a determines that a transverse line exists (S11, No), it again performs the process of S5, i.e., the process of selecting a transverse line using the procedure described above. On the other hand, if the epipolar planar image generating unit 14a determines that a transverse line does not exist (S11, Yes), it outputs a processing end notification signal to the depth recording unit 18.

[0088] When the depth recording unit 18 receives the processing end notification signal from the epipolar plane image generating unit 14a, it performs the processing in S12 to record the depth distribution DepthMap(j;m p ,Aα j +B+Ns j ) and terminate the process.

[0089] <Effects of the second embodiment> The output depth map (j;m p ,Aα j +B+Ns j ), it is possible to grasp the relative depth of the subject shown in each of the image data 41-1 to 41-J generated by imaging by each of the line sensors 20-1 to 20-J, i.e., the degree of distance between the satellite 2 and each of the subjects. Although the image processing device 10a cannot grasp the exact distance between the satellite 2 and the subject like the image processing device 10, it is possible to grasp the positional relationship between the satellite 2 and each of the subjects. Therefore, by using the image processing device 10a, it is possible to determine the position in the depth direction of the subject shown in the image data 41-1 to 41-J captured and generated by the existing imaging device 4, without adding any imaging equipment to the satellite 2. In addition, the depth distribution DepthMap(j;m p ,Aα j +B+Ns j), by setting an appropriate threshold value and dividing the area, it is possible to segment, for example, the ice and snow parts on the ground 100 and the cloud parts in the air, in the same way as the image processing device 10.

[0090] <Effects common to the first and second embodiments> The line sensors 20-1 to 20-J included in the imaging device 4 are imaging means that capture visible light. Imaging means that capture visible light images have higher spatial resolution than infrared sensors, LiDAR, etc., and therefore can achieve higher discrimination accuracy than when using infrared sensors, LiDAR, etc. Furthermore, the image processing devices 10, 10a generate epipolar plane image data to enable robust measurement of positional deviation due to parallax. Therefore, when there are differences in color between cloud portions displayed in the image data 41-1 to 41-J, it is possible to calculate the depth of each cloud portion, and therefore it is also possible to estimate the three-dimensional structure of the cloud surface.

[0091] <Modifications of the embodiment> In the image processing device 10, 10a, the epipolar planar image generating unit 14, 14a may normalize the brightness levels of the pixel values ​​included in the extracted line image data for each extracted line image data when extracting the extracted line image data from each of the image data 41-1 to 41-J. This normalization is a normalization that clarifies the lines that form the stripe pattern displayed in the epipolar planar image data. This normalization may be, for example, a normalization that makes the average brightness levels of each pixel included in the extracted line image data consistent across all extracted line image data.

[0092] The image processing device 10, 10a processes satellite image data obtained by capturing images using each of the J line sensors 20-1 to 20-J of the imaging device 4 provided on the satellite 2. In contrast, the image processing device 10, 10a is not limited to processing satellite image data, and may also process a plurality of image data generated by capturing images using each of the line sensors 20-1 to 20-J arranged so that their line-of-sight directions are different. In this case, the line sensors 20-1 to 20-J may be mounted on a flying airplane or a traveling vehicle and thus move, or the line sensors 20-1 to 20-J may be stationary and the subject may move. In other words, as long as each of the line sensors 20-1 to 20-J captures an image of a subject whose positional relationship with the line sensors 20-1 to 20-J changes over time, the line sensors 20-1 to 20-J may move, the subject may move, or both may move.

[0093] Although the imaging device 4 is described as including line sensors 20-1 to 20-J, the imaging device 4 may be configured to include one area sensor instead of line sensors 20-1 to 20-J. For example, if the area sensor has J rows and M columns of pixels, and this area sensor is considered to be J line sensors each having M pixels per row, i.e., J imaging means, then the configuration can be considered to be the same as when the imaging device 4 includes line sensors 20-1 to 20-J.

[0094] Furthermore, when the imaging device 4 includes one area sensor, the following processing may be performed using this one area sensor as one imaging means. That is, if this one imaging means is provided on the satellite 2 and there is an overlapping portion between image data obtained by capturing images at different times, a relative positional shift will occur in the image of the cloud 80 appearing in this overlapping portion. In this case, the depth of the subject can be detected even if the image data obtained by capturing images at different times using the area sensor is processed by the image processing device 10, 10a.

[0095] In the imaging device 4, each of the line sensors 20-1 to 20-J is provided with a bandpass filter that transmits light of different wavelength bands in order to capture visible light of different wavelength bands. Alternatively, some of the line sensors 20-1 to 20-J may be configured to capture visible light of the same wavelength band. Alternatively, all of the line sensors 20-1 to 20-J may be configured to capture visible light of the same wavelength band, in which case all of the line sensors 20-1 to 20-J do not need to be equipped with a bandpass filter.

[0096] Although the line sensors 20-1 to 20-J provided in the imaging device 4 are described as imaging means for capturing images in the wavelength band of visible light, they may also be imaging means for capturing images in wavelength bands other than the wavelength band of visible light.

[0097] 11, all imaging traces are selected as crossing lines, and in the processing of the image processing device 10a, all selectable imaging traces are selected as crossing lines. On the other hand, if the main purpose is to segment the cloud 80, the segmentation processing of the cloud 80 may be performed after the processing of S10, and the processing may be terminated when the segmentation of the cloud 80 is completed.

[0098] In the above, the processing of the image processing system 1, 1a has been described on the assumption that the direction in which the satellite 2 moves and the longitudinal direction of each of the line sensors 20-1 to 20-J are perpendicular to each other. However, the direction in which the satellite 2 moves and the longitudinal direction of each of the line sensors 20-1 to 20-J may intersect, rather than being perpendicular to each other. If they are not perpendicular to each other, the J pieces of image data obtained by capturing images with each of the line sensors 20-1 to 20-J will not simply be rectangular like the image data 41-1 to 41-3 shown in Figures 4 and 12. Even if they are not rectangular, this does not hinder processing by the image processing device 10, 10a, and ultimately the depth of the subject can be obtained.

[0099] 3, the line of sight direction of the line sensors 20-1 to 20-J is expressed by the angle between the line of sight direction and, for example, the direction of the optical axis of the optical system of the imaging device 4 as a reference direction, but the direction of the optical axis of the optical system of the imaging device 4 does not have to be the reference direction. For example, the vertical direction from the satellite 2 toward the ground 100 may be the reference direction.

[0100] In the above configuration, when the control device 3 outputs an imaging instruction signal to the imaging device 4, the control unit 21 receives the imaging instruction signal and supplies a repetitive pulse signal to each of the line sensors 20-1 to 20-J in parallel at regular intervals. Alternatively, the control unit 21 may supply a pulse signal to one of the line sensors 20-1 to 20-J at regular intervals, so that each of the line sensors 20-1 to 20-J captures an image the same number of times during a predetermined imaging time. In other words, the imaging device 4 may generate multiple pieces of image data with different imaging start and end timings, provided that the pieces of image data have the same size. Furthermore, the imaging device 4 may generate multiple pieces of image data with different sizes, provided that there is an overlapping portion between them. In other words, the number of pulse signals supplied to each of the line sensors 20-1 to 20-J during a predetermined imaging time may be different based on a single imaging instruction signal.

[0101] 11, the information acquisition unit 12 calculates, from the orbit information, the moving direction of the satellite 2 and the altitude of the satellite 2 at the generation time output by the image acquisition unit 11. Here, the generation time output by the image acquisition unit 11 is the time indicating the timing at which the control device 3 provides an imaging instruction signal to the imaging device 4. On the other hand, if there is a difference between the moving direction of the satellite 2 and the altitude of the satellite 2 at the beginning and end of a predetermined imaging time, and this difference cannot be ignored, the information acquisition unit 12 may calculate the moving direction of the satellite 2 and the altitude of the satellite 2 from the orbit information as follows.

[0102] For example, a predetermined imaging time is stored in advance in a storage area inside the information acquisition unit 12. When the information acquisition unit 12 acquires the generation time from the image acquisition unit 11, the information acquisition unit 12 sets the generation time to the time obtained by adding half of the predetermined imaging time to the acquired generation time. In this case, the information acquisition unit 12 calculates the direction of movement of the satellite 2 and the altitude of the satellite 2 when half of the predetermined imaging time has elapsed.

[0103] In response to this, when the information acquisition unit 12 acquires the generation time from the image acquisition unit 11, it adds a predetermined imaging time to the acquired generation time to calculate the completion time. The information acquisition unit 12 calculates the movement direction of the satellite 2 as the average direction of the movement direction of the satellite 2 calculated based on the generation time and the movement direction of the satellite 2 calculated based on the completion time. The information acquisition unit 12 may also calculate the altitude of the satellite 2 as the average altitude of the satellite 2 calculated based on the generation time and the altitude of the satellite 2 calculated based on the completion time.

[0104] Using these procedures, it is possible to obtain the average movement direction of satellite 2 during a specified imaging time and the altitude of satellite 2. Note that the information acquisition unit 12 may also calculate the movement direction of satellite 2 and the altitude of satellite 2 at any time between the generation time and the completion time.

[0105] Although the image superimposing unit 13 uses the WGS84 ellipsoid as the predetermined geodetic system, it may use a geodetic system other than the WGS84 ellipsoid.

[0106] Although the quantification units 16 and 16a use the Hough transform method as the predetermined image processing, methods other than the Hough transform method, such as template matching and optical flow methods, may also be used.

[0107] Although the image processing devices 10 and 10a are described as being provided in the building of the ground station 5, they may be installed in a location other than the building of the ground station 5 and connected to the ground station device 9 via, for example, a communication network. Furthermore, in the image processing systems 1 and 1a, the users of the satellite 2 and the ground station device 9 and the users of the image processing devices 10 and 10a may be different people who have no relationship with each other. Satellite image data, orbital information of the satellite 2, and information indicating the line-of-sight direction of each of the line sensors 20-1 to 20-J may be made public by the ground station device 9 connected to the Internet, for example, and each of the image processing devices 10 and 10a may obtain the data and information it requires from the ground station device 9 via the Internet. In this case, the satellite image data, the orbit information of satellite 2, and the information indicating the line of sight direction of each of line sensors 20-1 to 20-J do not have to be made public on the Internet by terrestrial station device 9 that communicates wirelessly with satellite 2, but may be a general server device that has acquired the satellite image data, the orbit information of satellite 2, and the information indicating the line of sight direction of each of line sensors 20-1 to 20-J by some means from terrestrial station device 9. Alternatively, the satellite image data, the orbit information of satellite 2, and the information indicating the line of sight direction of each of line sensors 20-1 to 20-J stored in terrestrial station device 9 may be copied to a storage device such as a hard disk, and the storage device may be connected to image processing devices 10, 10a, so that each of image processing devices 10, 10a acquires the data and information it needs from the connected storage device.

[0108] <Hardware configuration> FIG. 16 is a diagram illustrating an example of the hardware configuration of the image processing device 10, 10a shown in FIG. 10 and FIG. 15 according to the present disclosure. The image processing device 10, 10a according to the present disclosure is a computer including, for example, a central processing unit (CPU) 201, a random access memory (RAM) 202, a read-only memory (ROM) 203, an auxiliary storage device 204, an interface module 205, an input module 206, and an output module 207. The CPU 201, RAM 202, ROM 203, the auxiliary storage device 204, the interface module 205, the input module 206, and the output module 207 are interconnected by a bus 208. The auxiliary storage device 204 is, for example, a hard disk drive (HDD) or a solid state drive (SDD). The interface module 205 is, for example, a communication interface connected to a ground station device 9 provided in the image acquisition unit 11 and the information acquisition unit 12. The input module 206 is included in, for example, the image acquisition unit 11, and the output module 207 is included in, for example, the depth calculation unit 17 and the depth recording unit 18.

[0109] When an application program pre-stored in ROM 203 or auxiliary storage device 204 is executed by CPU 201, an image acquisition unit 11, an information acquisition unit 12, an image superposition unit 13, an epipolar plane image generation unit 14, 14a, a depth detection unit 15, 15a, a quantification unit 16, 16a, a depth calculation unit 17, and a depth recording unit 18 are configured, and internal storage space for each of the above-mentioned functional units is secured in RAM 202 or auxiliary storage device 204.

[0110] Third Embodiment An embodiment of the present disclosure will be described below with reference to the drawings. As shown in Fig. 17, an image processing device 300 includes image acquisition means 301 that acquires image data generated by imaging means arranged so as to have different line-of-sight directions, image superimposition means 302 that aligns and superimposes the image data so as to eliminate absolute positional deviations caused by different line-of-sight directions in the image data, epipolar plane image generation means 303 that selects a transverse line that crosses an overlapping portion of the superimposed image data according to the direction in which the ranges of the superimposed image data shift, and generates epipolar plane image data by arranging each of the image data of the portion where the selected transverse line and the overlapping portion overlap in an order determined by the magnitude of the gradient of the line-of-sight direction corresponding to each of the selected transverse line and the overlapping portion, and depth detection means 304 that detects the depth of the subject represented in the image data from a streak pattern represented in the epipolar plane image data.

[0111] As shown in Fig. 18, in the image processing device 300, the image acquisition means 301 acquires image data generated by imaging means arranged so as to have different line-of-sight directions (S301). The image superimposition means 302 aligns the image data acquired by the image acquisition means 301 to eliminate absolute positional deviations caused by different line-of-sight directions, and superimposes the image data (S302). The epipolar plane image generation means 303 selects a transverse line that crosses the overlapping portion of the superimposed image data according to the direction in which the ranges of the image data superimposed by the image superimposition means 302 shift (S303). The epipolar plane image generation means 303 generates epipolar plane image data by arranging the image data of the overlapping portion where the selected transverse line and the overlapping portion overlap in an order determined by the magnitude of the gradient of the corresponding line-of-sight direction (S304). The depth detection means 304 detects the depth of the subject appearing in the image data from the streak pattern appearing in the epipolar plane image data generated by the epipolar plane image generation means 303 (S305), and the process ends.

[0112] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0113] Some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0114] (Supplementary Note 1) An image processing device comprising: an image acquisition means (e.g., image acquisition unit 11) for acquiring image data generated by imaging means arranged so as to have different line-of-sight directions; an image superimposition means (e.g., image superimposition unit 13) for aligning and superimposing the image data so as to eliminate absolute positional deviations caused by the different line-of-sight directions in each of the image data; an epipolar plane image generation means (e.g., epipolar plane image generation unit 14, 14a) for selecting a transverse line that crosses an overlapping portion of the superimposed image data according to the direction in which the ranges of each of the image data in the superimposed state shift, and arranging each of the image data of the portion where the selected transverse line and the overlapping portion overlap in an order determined by the magnitude of the inclination of the line-of-sight direction corresponding to each of the image data; and a depth detection means (e.g., depth detection unit 15, 15a) for detecting the depth of a subject represented in the image data from a streak pattern represented in the epipolar plane image data.

[0115] (Appendix 2) The image processing device described in (Appendix 1), wherein there are a plurality of the imaging means, and each of the plurality of imaging means is arranged on a single moving object so that the magnitude of the tilt of the line of sight increases according to the order of arrangement of the imaging means, the order determined by the magnitude of the tilt of the line of sight of the imaging means is the order of arrangement of the imaging means, and the direction in which the range of each of the image data in the superimposed state shifts is derived from the direction of movement of the object.

[0116] (Appendix 3) The image processing device described in (Appendix 2), wherein the imaging means is a line sensor, the longitudinal direction of the imaging means intersects with the direction of movement of the object, and one imaging means repeatedly captures images at a predetermined imaging time while the object is moving, and multiple line image data generated for each repeated imaging are combined to generate one piece of image data.

[0117] (Appendix 4) The image processing device according to (Appendix 2) or (Appendix 3), wherein all or some of the imaging means capture images of the same wavelength band of visible light or different wavelength bands of visible light.

[0118] (Appendix 5) The image processing device described in any one of (Appendix 2) to (Appendix 4), wherein the epipolar plane image generating means acquires information indicating the movement direction of the object, identifies the positions of the image data corresponding to the center points of each of the pixels included in the multiple imaging means as imaging points, detects imaging traces which are the trajectories of the imaging points based on each of the identified imaging points and the acquired information indicating the movement direction, and selects one of the imaging traces common to the multiple imaging means from the detected imaging traces as the transverse line.

[0119] (Appendix 6) An image processing device described in any one of (Appendix 1) to (Appendix 5), wherein the line of sight direction is directed toward the ground, and the image superimposition means projects each of the image data onto a predetermined geodetic system as the alignment.

[0120] (Appendix 7) An image processing device described in any one of (Appendix 1) to (Appendix 6), wherein the depth detection means quantifies the slope of the lines forming the streak pattern and detects the depth of the subject based on the value obtained by the quantification.

[0121] (Appendix 8) The image processing device described in (Appendix 7), wherein the depth detection means acquires information indicating the line of sight and information indicating the altitude at which the image was captured, performs the quantification using the acquired information indicating the line of sight, and detects the depth of the subject based on the acquired information indicating the altitude and the value obtained by the quantification.

[0122] (Appendix 9) An image processing system comprising: an imaging means; and an image processing device, wherein the image processing device comprises: image acquisition means (e.g., image acquisition unit 11) that acquires image data generated by imaging means arranged so as to have different line-of-sight directions; image superimposition means (e.g., image superimposition unit 13) that aligns and superimposes the image data so as to eliminate absolute positional deviations that occur due to the different line-of-sight directions in each of the image data; epipolar plane image generation means (e.g., epipolar plane image generation unit 14, 14a) that selects a transverse line that crosses an overlapping portion of the superimposed image data according to a direction in which the ranges of each of the image data in the superimposed state shift, and arranges each of the image data of the portion where the selected transverse line and the overlapping portion overlap in an order determined by the magnitude of the gradient of the line-of-sight direction corresponding to each of the image data; and depth detection means (e.g., depth detection unit 15, 15a) that detects the depth of a subject represented in the image data from a streak pattern represented in the epipolar plane image data.

[0123] (Appendix 10) The image processing system described in (Appendix 9), wherein there are a plurality of the imaging means, and each of the plurality of imaging means is arranged on a moving object so that the magnitude of the tilt of the line of sight increases according to the order of arrangement of the imaging means, the order determined by the magnitude of the tilt of the line of sight of the imaging means is the order of arrangement of the imaging means, and the direction in which the range of each of the image data in the superimposed state shifts is derived from the direction of movement of the object.

[0124] (Appendix 11) The image processing system described in (Appendix 10), wherein the imaging means is a line sensor, the longitudinal direction of the imaging means intersects with the direction of movement of the object, and one imaging means repeatedly captures images at a predetermined imaging time while the object is moving, and multiple line image data generated for each repeated imaging is combined to generate one piece of image data.

[0125] (Appendix 12) An image processing system according to (Appendix 10) or (Appendix 11), wherein all or some of the imaging means capture images of the same wavelength band in visible light or different wavelength bands in visible light.

[0126] (Appendix 13) The image processing system described in any one of (Appendix 10) to (Appendix 12), wherein the epipolar plane image generation means acquires information indicating the movement direction of the object, identifies the positions of the image data corresponding to the center points of each of the pixels included in the multiple imaging means as imaging points, detects imaging traces, which are the trajectories of the imaging points, based on each of the identified imaging points and the acquired information indicating the movement direction, and selects, from the detected imaging traces, one of the imaging traces common to the multiple imaging means as the transverse line.

[0127] (Appendix 14) An image processing system described in any one of (Appendix 9) to (Appendix 13), wherein the line of sight direction is directed toward the ground, and the image superimposition means projects each of the image data onto a predetermined geodetic system as the alignment.

[0128] (Appendix 15) An image processing system described in any one of (Appendix 9) to (Appendix 14), wherein the depth detection means quantifies the slope of the lines forming the streak pattern and detects the depth of the subject based on the value obtained by the quantification.

[0129] (Appendix 16) The depth detection means acquires information indicating the line of sight and information indicating the altitude at which the image was captured, performs the quantification using the acquired information indicating the line of sight, and detects the depth of the subject based on the acquired information indicating the altitude and the value obtained by the quantification (image processing system described in Appendices 15).

[0130] (Supplementary Note 16) An image processing method comprising: acquiring image data generated by imaging means arranged so as to have different line-of-sight directions; superimposing the acquired image data by aligning the image data so as to eliminate absolute positional deviations that arise due to the different line-of-sight directions in each of the acquired image data; selecting a transverse line that crosses an overlapping portion of the superimposed image data according to the direction in which the ranges of each of the superimposed image data shift; arranging each of the image data in the portion where the selected transverse line and the overlapping portion overlap in an order determined by the magnitude of the gradient of the line-of-sight direction corresponding to each of the image data; and detecting the depth of a subject represented in the image data from a streak pattern represented in the generated epipolar plane image data.

[0131] (Appendix 17) The image processing method described in (Appendix 16), wherein there are a plurality of the imaging means, and each of the plurality of imaging means is arranged on a single moving object so that the magnitude of the tilt of the line of sight increases according to the order of arrangement of the imaging means, the order determined by the magnitude of the tilt of the line of sight of the imaging means is the order of arrangement of the imaging means, and the direction in which the range of each of the image data in the superimposed state shifts is derived from the direction of movement of the object.

[0132] (Appendix 18) The image processing method described in (Appendix 17), wherein the imaging means is a line sensor, the longitudinal direction of the imaging means intersects with the direction of movement of the object, and one imaging means repeatedly captures images at a predetermined imaging time while the object is moving, and multiple line image data generated for each repeated imaging are combined to generate one piece of image data.

[0133] (Appendix 19) An image processing method according to (Appendix 17) or (Appendix 18), wherein all or part of the imaging means captures the same wavelength band of visible light or different wavelength bands of visible light.

[0134] (Appendix 20) An image processing method according to any one of (Appendix 17) to (Appendix 19), comprising acquiring information indicating the direction of movement of the object, identifying the positions of the image data corresponding to the center points of each of the pixels included in the plurality of imaging means as imaging points, detecting imaging traces, which are the trajectories of the imaging points, based on each of the identified imaging points and the acquired information indicating the direction of movement, and selecting, from the detected imaging traces, one of the imaging traces common to the plurality of imaging means as the transverse line.

[0135] (Appendix 21) An image processing method described in any one of (Appendix 16) to (Appendix 20), wherein the line of sight direction is directed toward the ground, and the alignment involves projecting each of the image data onto a predetermined geodetic system.

[0136] (Appendix 22) An image processing method described in any one of (Appendix 16) to (Appendix 21), wherein the depth detection means quantifies the slope of the lines forming the streak pattern and detects the depth of the subject based on the value obtained by the quantification.

[0137] (Appendix 23) The image processing method described in (Appendix 22), wherein the depth detection means acquires information indicating the line of sight and information indicating the altitude at which the image was captured, performs the quantification using the acquired information indicating the line of sight, and detects the depth of the subject based on the acquired information indicating the altitude and the value obtained by the quantification.

[0138] (Appendix 24) A program for causing a computer to function as an image acquisition means for acquiring image data generated by imaging means arranged so as to have different line-of-sight directions; an image superimposition means for aligning the image data so as to eliminate absolute positional deviations that arise due to the different line-of-sight directions in each of the image data and superimposing the image data; an epipolar plane image generation means for selecting a transverse line that crosses an overlapping portion of the superimposed image data according to the direction in which the range of each of the image data in the superimposed state shifts, and arranging each of the image data in the portion where the selected transverse line and the overlapping portion overlap in an order determined by the magnitude of the inclination of the line-of-sight direction corresponding to each of the image data; and a depth detection means for detecting the depth of a subject represented in the image data from a streak pattern represented in the epipolar plane image data.

[0139] (Appendix 25) The program described in (Appendix 24), wherein there are a plurality of the imaging means, and each of the plurality of imaging means is arranged on a moving object so that the magnitude of the tilt of the line of sight increases according to the order of arrangement of the imaging means, the order determined by the magnitude of the tilt of the line of sight of the imaging means is the order of arrangement of the imaging means, and the direction in which the range of each of the image data in the superimposed state shifts is derived from the direction of movement of the object.

[0140] (Appendix 26) The program described in (Appendix 25), wherein the imaging means is a line sensor, the longitudinal direction of the imaging means intersects with the direction of movement of the object, and one imaging means repeatedly captures images at a predetermined imaging time while the object is moving, and multiple line image data generated for each repeated imaging are combined to generate one image data.

[0141] (Appendix 27) The program described in (Appendix 25) or (Appendix 26), wherein all or part of the imaging means images the same wavelength band in visible light or different wavelength bands in visible light.

[0142] (Appendix 28) The epipolar plane image generating means acquires information indicating the movement direction of the object, identifies the positions of the image data corresponding to the center points of each of the pixels included in the multiple imaging means as imaging points, detects imaging traces, which are the trajectories of the imaging points, based on each of the identified imaging points and the acquired information indicating the movement direction, and selects, from the detected imaging traces, one of the imaging traces common to the multiple imaging means as the transverse line.

[0143] (Appendix 29) A program described in any one of (Appendix 24) to (Appendix 28), wherein the line of sight direction is directed toward the ground, and the image superimposition means projects each of the image data onto a predetermined geodetic system as the alignment.

[0144] (Appendix 30) A program described in any one of (Appendix 24) to (Appendix 29), wherein the depth detection means quantifies the slope of the lines forming the streak pattern and detects the depth of the subject based on the value obtained by the quantification.

[0145] (Appendix 31) The depth detection means acquires information indicating the line of sight and information indicating the altitude at which the image was captured, performs the quantification using the acquired information indicating the line of sight, and detects the depth of the subject based on the acquired information indicating the altitude and the value obtained by the quantification. (Appendix 30) The program described in [Explanation of symbols]

[0146] 1. Image processing system 2 satellites 3. Control device 4. Imaging device 5. Ground Station 9 Ground station equipment 10 Image processing device 11 Image acquisition unit 12 Information acquisition department 13 Image superimposition unit 14 Epipolar plane image generation unit 15 Depth detection unit 16 Quantification Department 17 Depth calculation section 20-1~20-J Line Sensor 21 Control section 22 Lens 30-j-1~30-jM CCD element

Claims

1. image acquisition means for acquiring image data generated by image capture by image capture means arranged so that the line of sight directions are different; an image superimposing means for superimposing the image data by aligning the image data so as to eliminate any absolute positional deviation caused by differences in the line of sight directions between the image data; an epipolar plane image generating means for selecting a transverse line that crosses an overlapping portion of the superimposed image data in accordance with a direction in which the ranges of the image data in the superimposed state shift, and arranging each of the image data of the portion where the selected transverse line and the overlapping portion overlap in an order determined by the magnitude of the inclination of the line of sight direction corresponding to each of the image data, thereby generating epipolar plane image data; a depth detection means for detecting the depth of a subject appearing in the image data from a streak pattern appearing in the epipolar plane image data; An image processing device comprising:

2. There are a plurality of the imaging means, each of the plurality of imaging means is arranged on one moving object such that the magnitude of the inclination of the line of sight increases in the order of arrangement of the imaging means; the order determined by the magnitude of the inclination of the line of sight direction of the imaging means is the arrangement order of the imaging means, The direction in which the ranges of the image data in the superimposed state shift is derived from the direction of movement of the object. The image processing device according to claim 1 .

3. the imaging means is a line sensor, a longitudinal direction of the imaging means intersecting with a direction of movement of the object; One of the imaging means repeatedly captures images of the object for a predetermined imaging time while the object is moving, and a plurality of line image data generated for each of the repeated imaging operations are compiled to generate one image data set. The image processing device according to claim 2 .

4. All or some of the imaging means capture images of the same wavelength band in visible light or different wavelength bands in visible light. The image processing device according to claim 2 .

5. The line of sight is directed toward the ground, The image superimposing means As the registration, each of the image data is projected onto a predetermined geodetic system. The image processing device according to claim 1 .

6. The depth detection means quantifying the gradient of the lines forming the streak pattern, and detecting the depth of the subject based on the value obtained by the quantification; The image processing device according to any one of claims 1 to 5.

7. The depth detection means acquiring information indicating the line of sight direction and information indicating an altitude at which the image was captured, performing the quantification using the acquired information indicating the line of sight direction, and detecting the depth of the subject based on the acquired information indicating the altitude and a value obtained by the quantification; The image processing device according to claim 6 .

8. An imaging means; an image processing device, The image processing device includes: image acquisition means for acquiring image data generated by the imaging means arranged so that the line of sight directions are different from each other; an image superimposing means for superimposing the image data by aligning the image data so as to eliminate any absolute positional deviation caused by differences in the line of sight directions between the image data; an epipolar plane image generating means for selecting a transverse line that crosses an overlapping portion of the superimposed image data in accordance with a direction in which the ranges of the image data in the superimposed state shift, and arranging each of the image data of the portion where the selected transverse line and the overlapping portion overlap in an order determined by the magnitude of the inclination of the line of sight direction corresponding to each of the image data, thereby generating epipolar plane image data; a depth detection means for detecting the depth of a subject appearing in the image data from a streak pattern appearing in the epipolar plane image data; An image processing system comprising:

9. Acquiring image data generated by imaging means arranged so that the line of sight directions are different; superimposing the image data by aligning the image data so as to eliminate absolute positional deviations that occur due to differences in the line of sight directions in each of the acquired image data; selecting a crossing line that crosses an overlapping portion of the superimposed image data according to a direction in which the ranges of the superimposed image data are shifted; generating epipolar plane image data by arranging each of the image data of the portion where the selected transverse line and the overlapping portion overlap in an order determined by the magnitude of the inclination of the corresponding line of sight direction; detecting a depth of the subject appearing in the image data from a streak pattern appearing in the generated epipolar plane image data; Image processing methods.

10. Computer, image acquisition means for acquiring image data generated by image capture by imaging means arranged so that the line of sight directions are different; an image superimposing means for superimposing the image data by aligning the image data so as to eliminate any absolute positional deviation caused by differences in the line of sight directions between the image data; an epipolar plane image generating means for selecting a transverse line that crosses an overlapping portion of the superimposed image data in accordance with a direction in which the ranges of each of the superimposed image data shift, and arranging each of the image data of the portion where the selected transverse line and the overlapping portion overlap in an order determined by the magnitude of the inclination of the line of sight direction corresponding to each of the image data, thereby generating epipolar plane image data; a depth detection means for detecting the depth of a subject appearing in the image data from a streak pattern appearing in the epipolar plane image data; A program to function as a

Images