Image processing device, artificial satellite, image processing system, image processing method, and program

Abstract

The image processing device includes an acquisition unit that acquires a plurality of area images each consisting of M×N pixels, where M pixels are in a first direction corresponding to the direction of movement of the moving body and N pixels are in a second direction intersecting the first direction, and acquires a plurality of line images each consisting of n pixels in the first direction and n×L pixels, where L pixels are in the second direction, and are imaged by a second imaging device mounted on the moving body; an identification unit that identifies the pixel shift amount in the first direction and the second direction between the plurality of area images based on the plurality of area images; and a generation unit that generates transmission data for transmitting the plurality of line images and pixel shift amount information indicating the pixel shift amount via a communication unit to an image generation device that generates a concatenated image by concatenating a plurality of line images whose positions in the first direction and the second direction are adjusted using the pixel shift amount and that are arranged in the first direction.

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an artificial satellite, an image processing system, an image processing method, and a program. [Background technology]

[0002] Non-Patent Document 1 discloses a method for analogically deriving pixel shifts between images of an area sensor using a Fourier optical system. [Prior art document] [Patent documents] [Non-Patent Document 1] K. Janschek et al. SmartScan: a robust pushbroom imaging concept for moderate spacecraft attitude stability, International Conference on Space Optics ICSO 2006, 27-30 June 2006. General Disclosure

[0003] An image processing device according to one aspect of the present invention may include an acquisition unit that acquires a plurality of area images, each composed of M×N pixels (M and N are integers of 2 or greater), captured by a first imaging device mounted on a moving object, with M pixels in a first direction corresponding to the moving direction of the moving object and N pixels in a second direction intersecting the first direction, and acquires a plurality of line images, each composed of n pixels in the first direction and n×L pixels (n is an integer of 1 or greater, and L is an integer of 3 or greater than M and N), captured by a second imaging device mounted on the moving object. The image processing device may include an identification unit that identifies pixel shift amounts between the plurality of area images in the first direction and the second direction based on the plurality of area images. The image processing device may include a generation unit that generates transmission data for transmitting the plurality of line images and pixel shift amount information indicating the pixel shift amount via a communication unit to an image generation device that generates a concatenated image by concatenating the plurality of line images, whose positions in the first direction and the second direction are adjusted using the pixel shift amount.

[0004] In any one of the image processing devices, the specifying unit may specify the pixel shift amount by comparing the plurality of area images with each other by template matching.

[0005] In any of the image processing devices, the plurality of area images may include a first area image and a second area image captured subsequent to the first area image. The identification unit may identify a second partial image having a highest correlation among the plurality of second partial images as a target partial image by performing template matching between a first partial image in a first region within the first area image and each of a plurality of second partial images included in a search region larger than the first region, the second partial image including a second region in the second area image shifted from the first region by a first number of pixels in the first direction and a second number of pixels in the second direction, and identify the pixel shift amount based on the first partial image and the target partial image.

[0006] In any of the image processing devices, the identification unit may derive the degree of correlation between the first partial image and each of the multiple second partial images from a cross-correlation function obtained by convolving the first partial image with each of the multiple second partial images in real space.

[0007] In any of the image processing devices, the first number of pixels and the second number of pixels may be determined based on a moving direction and a moving speed of the moving object, and a moving direction and a moving speed of the subject relative to the first imaging device.

[0008] In any of the image processing devices, the identification unit may start searching from a second partial image whose central pixel is a pixel located at the center of the second region among the plurality of second partial images, and identify the target partial image by template matching the second partial images with the first partial image in order, starting from the second partial image closest to the second partial image.

[0009] In any of the image processing devices, the identification unit may identify the pixel shift amount in sub-pixel units from the positional relationship between the center of the first partial image and the center of gravity derived from the cubed or fourth power of each normalized value of the correlation degree with the target partial image and the correlation degree with each of a plurality of surrounding partial images whose centers are within a predetermined range of pixels from the center of the target partial image.

[0010] In any of the image processing devices, the identification unit may identify the fourth partial image with the highest correlation among the multiple fourth partial images as another target partial image by template matching a third partial image in a third area different from the first area in the first area image with each of multiple fourth partial images included in a search area larger than the third area, including a fourth area shifted from the third area in the second area image by the first number of pixels in the first direction and the second number of pixels in the second direction, and identify the amount of pixel shift based on the first partial image, the target partial image, the third partial image, and the other target partial image.

[0011] In any of the image processing devices, the identification unit may identify the pixel shift amount between the first area image and the second area image based on statistical values of the pixel shift amount based on the first partial image and the target partial image and the pixel shift amount based on the third partial image and the other target partial image.

[0012] In any of the image processing devices, the first imaging device may include an area sensor that captures the plurality of area images, and the second imaging device may include a line sensor or a time delay integration (TDI) sensor that captures the plurality of line images.

[0013] In any of the image processing devices, the moving body may be an aircraft or an artificial satellite.

[0014] An artificial satellite according to one aspect of the present invention may include any one of the image processing devices, the first imaging device, the second imaging device, and the communication unit that transmits the transmission data to the image generation device.

[0015] An image processing system according to one aspect of the present invention may include the satellite and the image generating device that receives the transmission data and generates the combined image.

[0016] An image processing method according to one aspect of the present invention may include the steps of acquiring a plurality of area images, each composed of M×N pixels (M and N are integers of 2 or greater), where M pixels are in a first direction corresponding to the moving direction of the moving object and N pixels are in a second direction intersecting the first direction, captured by a first imaging device mounted on the moving object, and acquiring a plurality of line images, each composed of n pixels in the first direction and L pixels in the second direction (n×L pixels), where n is an integer of 1 or greater and L is an integer of 3 or greater and greater than M and N), captured by a second imaging device mounted on the moving object. The image processing method may include the step of determining pixel shift amounts between the plurality of area images in the first direction and the second direction, based on the plurality of area images. The image processing method may include the step of generating transmission data for transmitting the plurality of line images and pixel shift amount information indicating the pixel shift amounts via a communication unit to an image generating device that generates a concatenated image by concatenating the plurality of line images, whose positions in the first direction and the second direction are adjusted using the pixel shift amounts.

[0017] A program according to one aspect of the present invention, when executed by a computer, may cause the computer to function as an acquisition unit that acquires a plurality of area images captured by a first imaging device mounted on a moving object, each of which is composed of M×N pixels (M and N are integers of 2 or greater), where M pixels are in a first direction corresponding to the moving direction of the moving object and N pixels are in a second direction intersecting the first direction, and acquires a plurality of line images captured by a second imaging device mounted on the moving object, each of which is composed of n pixels in the first direction and n×L pixels (n is an integer of 1 or greater, and L is an integer of 3 or greater than M and N). The program may cause the computer to function as an identification unit that identifies pixel shift amounts between the plurality of area images in the first direction and the second direction, based on the plurality of area images. The program may also cause the computer to function as a generation unit that generates transmission data for transmitting the plurality of line images and pixel shift amount information indicating the pixel shift amount via a communication unit to an image generation device that generates a concatenated image by concatenating the plurality of line images whose positions in the first direction and the second direction are adjusted using the pixel shift amount.

[0018] The above summary of the invention does not list all of the features of the present invention, and subcombinations of these features may also be inventions. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of an image processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of functional blocks of the image processing apparatus according to the present embodiment. [Figure 3A] FIG. 2 is a diagram for explaining an area image and a partial image. [Figure 3B] FIG. 2 is a diagram for explaining an area image and a partial image. [Figure 4A]FIG. 10 is a diagram showing the simulation results of the estimation error distribution in the vertical direction (AT direction) when the pixel shift amount is estimated by template matching (TM) and phase-only correlation (POC). [Figure 4B] FIG. 10 is a diagram showing the simulation results of the estimation error distribution in the horizontal direction (CT direction) when the pixel shift amount is estimated by template matching (TM) and phase-only correlation (POC). [Figure 5] FIG. 10 is a diagram showing an example of simulation results of the average estimation error and standard deviation when pixel shift amounts are estimated by template matching (TM) and phase-only correlation (POC), respectively. [Figure 6] 10A and 10B are diagrams showing the average error and standard deviation of pixel shift amounts derived by parabolic fitting, centroid calculation using linear values, and centroid calculation using cube values. [Figure 7A] FIG. 10 is a diagram showing distributions of x-direction error and y-direction error of pixel shift amounts derived by parabolic fitting. [Figure 7B] 10A and 10B are diagrams illustrating distributions of x-direction error and y-direction error of pixel shift amounts derived by centroid calculation using first-power values. [Figure 7C] FIG. 10 is a diagram showing the distribution of x-direction error and y-direction error of pixel shift amount derived by centroid calculation using cubed values. [Figure 8A] This figure shows the simulation results of deriving pixel shift amounts by parabolic fitting, centroid calculation using a first power value, and centroid calculation using a third power value, using images with different sub-pixel amounts shifted in the horizontal direction (x-axis direction). [Figure 8B] This figure shows the simulation results of deriving pixel shift amounts by parabolic fitting, centroid calculation using a first power value, and centroid calculation using a third power value, using images with different sub-pixel amounts shifted in the horizontal direction (x-axis direction). [Figure 8C] This figure shows the simulation results of deriving pixel shift amounts by parabolic fitting, centroid calculation using a first power value, and centroid calculation using a third power value, using images with different sub-pixel amounts shifted in the horizontal direction (x-axis direction). [Figure 8D] This figure shows the simulation results of deriving pixel shift amounts by parabolic fitting, centroid calculation using a first power value, and centroid calculation using a third power value, using images with different sub-pixel amounts shifted in the horizontal direction (x-axis direction). [Figure 8E] This figure shows the simulation results of deriving pixel shift amounts by parabolic fitting, centroid calculation using a first power value, and centroid calculation using a third power value, using images with different sub-pixel amounts shifted in the horizontal direction (x-axis direction). [Figure 8F] This figure shows the simulation results of deriving pixel shift amounts by parabolic fitting, centroid calculation using a first power value, and centroid calculation using a third power value, using images with different sub-pixel amounts shifted in the horizontal direction (x-axis direction). [Figure 9A] FIG. 10 is a diagram showing the average estimation error and standard deviation in the x direction of pixel shift amounts obtained by centroid calculations performed while changing the value of the power of the normalized correlation function. [Figure 9B] FIG. 10 is a diagram showing the average estimation error and standard deviation in the x direction of pixel shift amounts obtained by centroid calculations performed while changing the value of the power of the normalized correlation function. [Figure 10] FIG. 2 is a diagram showing an example of a movement trajectory of an artificial satellite. [Figure 11] FIG. 10 is a diagram showing an example of a connected image obtained by connecting a plurality of line images. [Figure 12] 10 is a flowchart showing an example of an image transmission procedure of the image processing device 0. [Figure 13] FIG. 2 illustrates an example of a hardware configuration. DETAILED DESCRIPTION OF THE INVENTION

[0020] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0021] FIG. 1 is a diagram illustrating an example of a system configuration of an image processing system 300 according to this embodiment. The image processing system 300 includes an artificial satellite 10 and an image generation device 200. The artificial satellite 10 moves in space. The artificial satellite 10 may move in a predetermined orbit around the Earth. The artificial satellite 10 is an example of a mobile object. In this embodiment, the artificial satellite 10 is described as an example of a mobile object. However, the mobile object may also be an aircraft such as an unmanned aerial vehicle, a vehicle such as an automobile, or a ship. The artificial satellite 10 captures images of the Earth's surface while moving in orbit and provides the captured images to the image generation device 200. The image generation device 200 generates an image showing the Earth's surface by combining multiple images provided by the artificial satellite 10.

[0022] The artificial satellite 10 includes an image processing device 100. FIG.

[0023] The image processing device 100 includes a control unit 110, a storage unit 120, an area camera 130, a TDI camera 140, and a communication unit 150.

[0024] The area camera 130 captures multiple area images each consisting of M×N pixels (M and N are integers of 2 or more), where M pixels are in a first direction (vertical direction) corresponding to the moving direction of the satellite 10 and N pixels are in a second direction (horizontal direction) intersecting the first direction. The area camera 130 may have a low-resolution image sensor. The area camera 130 may capture an area image consisting of, for example, 100M pixels (1280×960). Each of M and N may be an integer of 2 or more and 1500 or less.

[0025] The TDI camera 140 captures a plurality of line images each consisting of n pixels in a first direction (vertical direction) corresponding to the direction of movement of the satellite 10 and L pixels in a second direction (horizontal direction) intersecting the first direction, where L is an integer greater than or equal to 1 and L is an integer greater than M and N and greater than 3. The TDI camera 140 may have a time delay integration (TDI) sensor. The TDI camera 140 may capture a line image of, for example, 1 x 12288 pixels. n may be an integer between 1 and 256, and L may be an integer greater than or equal to 4000, 6000, or 10,000.

[0026] The image processing device 100 may be provided with a line camera having a line sensor instead of the TDI camera 140. The TDI sensor captures a line image in which one pixel is constructed by adding together a plurality of pixels arranged in a first direction (vertical direction), and therefore can capture a high-quality image with a higher SNR (signal-to-noise ratio) than the line sensor.

[0027] When the artificial satellite 10 moves precisely and with high attitude stability on an orbit along a movement direction corresponding to the first direction, a relatively high-quality joined image showing the Earth's surface can be obtained by arranging and connecting line images in the first direction. This method is called the pushbroom method. However, in reality, the artificial satellite 10 may deviate from the orbit along the movement direction or deviate from the target attitude. Therefore, the image obtained by arranging and connecting line images in the first direction may be distorted.

[0028] On the other hand, there is also a method that does not use a TDI sensor, but instead uses a high-resolution area sensor to overlay high-resolution images to generate a single image. This method is called the Push Frame method. However, the process of comparing and overlaying high-resolution images is burdensome and requires a high-performance computer. Using a high-performance computer increases power consumption and weight. Therefore, for example, if the satellite 10 is a small satellite, it is not desirable to install such a heavy, power-hungry computer. Alternatively, it is possible to transmit images from the satellite 10 to a device on the ground, such as the image generation device 200, and have the images compared and overlaid by the device on the ground. However, because high-resolution images have a large data volume, the communication load increases when transmitting the images from the satellite 10 to a device on the ground, such as the image generation device 200.

[0029] Therefore, in this embodiment, the image processing device 100 identifies the pixel shift amount between the images by comparing low-resolution images captured by the area camera 130, and transmits pixel shift amount information indicating the pixel shift amount and multiple line images captured by the TDI camera 140 to the terrestrial image generation device 200. The image generation device 200 adjusts the position in the second direction (horizontal direction) using the pixel shift amount, and generates a concatenated image by concatenating multiple line images arranged in the first direction (vertical direction). Since the pixel shift amount is identified by comparing low-resolution images, the processing load on the image processing device 100 is reduced. Furthermore, the data transmitted to the terrestrial image generation device 200 includes pixel shift amount information, which is a relatively small amount of data, in addition to multiple line images, so an increase in the communication load when transmitting information from the artificial satellite 10 to the terrestrial image generation device 200 can be reduced.

[0030] The control unit 110 includes an acquisition unit 112, an identification unit 114, and a generation unit 116. The control unit 110 may be configured by a microprocessor such as a CPU or MPU, a microcontroller such as an MCU, or hardware such as an FPGA.

[0031] The acquisition unit 112 acquires a plurality of area images captured by the area camera 130 and a plurality of line images captured by the TDI camera 140. The acquisition unit 112 may acquire two or more line images between acquiring a first area image and acquiring a subsequent second area image.

[0032] The specifying unit 114 specifies the amount of pixel shift in the horizontal and vertical directions between the multiple area images based on the multiple area images. The specifying unit 114 may specify the amount of pixel shift by comparing the pixels of the multiple area images. The specifying unit 114 may specify the amount of pixel shift by comparing the multiple area images by template matching.

[0033] 3A and a second area image as shown in FIG. 3B that is captured subsequent to the first area image 401. The identification unit 114 may perform template matching between a first partial image in a first region 420a in the first area image 401 and each of a plurality of second partial images included in a search region 424a larger than the first region 420a, the search region 424a including a second region 422a having a center 423a shifted from a center 421a of the first region 420a in the second area image 402 by a first number of pixels 430a in a first direction (vertical direction) and by a second number of pixels 432a in a second direction (horizontal direction), thereby identifying a second partial image having the highest correlation among the plurality of second partial images as a target partial image, and may identify pixel shift amounts in the horizontal and vertical directions based on the first partial image and the target partial image. The first pixel number 430a and the second pixel number 432a are determined based on the moving direction and moving speed of the satellite 10, which is a moving body, and the moving direction and moving speed of the Earth's surface, which is the subject of the area camera 130. The moving direction and moving speed of the Earth's surface are determined by the direction of rotation of the Earth relative to the area camera 130, the speed of the Earth's rotation, and the speed of the satellite 10 relative to the ground.

[0034] The identification unit 114 may derive the degree of correlation between the first partial image and each of the multiple second partial images from a cross-correlation function obtained by convolving the first partial image with each of the multiple second partial images in real space, and identify the second partial image with the highest degree of correlation.

[0035] The artificial satellite 10 moves at a constant speed on a predetermined orbit. Therefore, it is highly likely that the position of a subject in the first area image 401 in the second area image 402 is near a position shifted from the position of the subject in the first area image 401 by a first number of pixels 430 in a first direction (vertical direction) and a second number of pixels 432 in a second direction (horizontal direction).

[0036] Therefore, the identification unit 114 may start searching from a second partial image whose central pixel is a pixel located at the center 423 of the second region 422 (search region) among the plurality of second partial images, and identify the target partial image by performing template matching with the first partial image in order from the second partial image closest to the second partial image, for example, clockwise or counterclockwise. This allows the second partial image with the highest correlation to be identified more quickly, and reduces the processing load on the control unit 110.

[0037] The identification unit 114 may derive correlations for multiple regions within the area image and identify the pixel shift amounts relative to the area image from the pixel shift amounts identified for each of the multiple regions. For example, the area image may include a region with little change in feature values, such as a forest, lake, or ocean. For such regions, it may be difficult for the identification unit 114 to accurately identify the pixel shift amounts using template matching. In such cases, it may be preferable for the identification unit 114 to identify pixel shift amounts for multiple regions, exclude outliers, and finally identify the pixel shift amounts relative to the area image. Furthermore, the identification unit 114 may identify the pixel shift amounts relative to the area image by deriving statistics of the pixel shift amounts identified for each of the multiple regions. The statistics may be, for example, an average value, a maximum value, a minimum value, or a variance.

[0038] The identification unit 114 may, for example, perform template matching between a third partial image in a third region 420b different from the first region 420a in the first area image 401 and each of multiple fourth partial images included in a search region 424b larger than the third region 420b, including a fourth region 422b having a center 423b shifted from the center 421b of the third region 420b in the second area image 402 by a first number of pixels 430b in a first direction (vertical direction) and a second number of pixels 432b in a second direction (horizontal direction), thereby identifying the fourth partial image with the highest correlation among the multiple fourth partial images as another target partial image, and may identify the amount of pixel shift based on the first partial image, the target partial image, the third partial image, and the other target partial images.

[0039] The identification unit 114 may further identify the sixth partial image having the highest correlation among the sixth partial images as a further target partial image by template matching a fifth partial image in a fifth region 420c different from the first region 420a and the third region 420c in the first area image 401 with each of a plurality of sixth partial images included in a search region 424c larger than the fifth region 420c, the sixth region 422c having a center 423c shifted from the center 421c of the fifth region 420c in the second area image 402 by a first number of pixels 430c in a first direction (vertical direction) and a second number of pixels 432c in a second direction (horizontal direction), and may identify the pixel shift amount further based on the fifth partial image and the further target partial image. The determination unit 114 may determine the pixel shift amount for the area image based on statistical values of the pixel shift amount determined from the first partial image and the target partial image, the pixel shift amount determined from the third partial image and another target partial image, and the pixel shift amount determined from the fifth partial image and yet another target partial image.

[0040] The determination unit 114 may determine the pixel shift amounts in the horizontal and vertical directions in sub-pixel units from the positional relationship between the center of the first partial image and the center of gravity derived from values obtained by cubed or fourth power normalizing the correlation degree with respect to the target partial image and the correlation degree with respect to each of multiple peripheral partial images whose centers are within a predetermined pixel range from the center of the target partial image. The pixel number range may be eight pixels surrounding the central pixel of the target partial image, that is, a 3 × 3 pixel number range centered on the central pixel of the target partial image.

[0041] In addition to the template matching described above, image processing techniques that the identification unit 114 uses to identify the pixel shift amounts in the horizontal and vertical directions between area images include feature point matching and phase-only correlation. However, it is preferable that the image processing device 100 installed on the artificial satellite 10 has as little processing load and power consumption as possible.

[0042] Feature point matching cannot correctly identify the image shift amount unless there is a feature amount specific to a local area, and the calculation cost is high. The subject captured by the area camera 130 is the ground surface, and may be, for example, a forest. In the case of an image having similar features over a wide range like this, there is no feature amount specific to the local area. Therefore, when feature point matching is adopted, the processing load on the image processing device 100 increases, which is not preferable.

[0043] Phase-only correlation is a method for matching images using the phase spectrum after Fourier transforming each of the images to be compared. In phase-only correlation, images are matched by focusing only on the phase component after Fourier transform, and the brightness (amplitude) information of the images is normalized.

[0044] Here, it is preferable that the image sensor provided in the area camera 130 mounted on the artificial satellite 10 is lightweight and small. Therefore, a small image sensor size is preferable. On the other hand, in order to match images, the images to be compared must contain the same subject (features). When the image sensor size is small, the interval between images captured by the area camera 130 must be relatively short, taking into account the speed of the artificial satellite 10. The interval is, for example, several milliseconds to several tens of milliseconds. Images captured at such short intervals are unlikely to have a large change in brightness. In other words, the correlation between brightness is high between images. Considering this point, the accuracy of estimating the amount of image shift decreases when using phase-only correlation, which normalizes brightness information.

[0045] 4A and 4B show examples of simulation results of the estimation error distribution when pixel shift amounts are estimated using template matching (TM) and phase-only correlation (POC), respectively. In FIGS. 4A and 4B, "POC nominal" indicates a case where one of the images to be compared is shifted in the vertical direction (AT direction) by a number of pixels that takes into account the movement of the satellite 10 before matching. On the other hand, "POC" indicates a case where one of the images to be compared is matched without being shifted in the vertical direction (AT direction) by a number of pixels that takes into account the movement of the satellite 10. FIG. 4A shows the estimation error distribution in the vertical direction (AT direction), and FIG. 4B shows the estimation error portion in the horizontal direction (CT direction).

[0046] Figure 5 shows an example of simulation results for the average estimation error and standard deviation when estimating pixel shifts using template matching (TM) and phase-only correlation (POC). The outliers in Figure 5 indicate the total number of results where the pixel shift is one pixel or more.

[0047] The simulation results shown in Figures 4A, 4B, and 5 were obtained by running 500 trials using two images shifted by 0.2 pixels horizontally and 0.3 pixels vertically as the two images to be compared, with different parts of the image used as the template image.

[0048] In phase-only correlation, the two images to be compared are Fourier transformed, and the two phase spectra obtained are combined to derive a normalized cross-correlation power spectrum. This result is then subjected to an inverse Fourier transform to obtain a map image of the correlation degree. The center of gravity is calculated from the correlation degree of a 5x5 pixel area surrounding the pixel with the maximum correlation degree in this map image, and the pixel shift amount between the two images is estimated.

[0049] On the other hand, template matching derives the correlation value by convolving two images in real space while shifting one of the images. Then, the correlation values derived for each position within a 3x3 pixel region centered on the pixel showing the highest correlation are normalized, and the center of gravity is calculated from the cubed values to estimate the pixel shift between the two images. While template matching also involves convolving two images in frequency space, this method is preferable because it limits the range over which the correlation is calculated, and therefore requires less computational cost than convolution in frequency space.

[0050] From the simulation results shown in FIGS. 4A, 4B, and 5, it can be seen that template matching (TM) has the smallest error and is the most preferable method for estimating the pixel shift amount.

[0051] In addition to the centroid calculation described above, methods for estimating pixel shift amounts on a sub-pixel basis in template matching include methods such as parabolic fitting. It is also possible to use a linear calculation for the centroid calculation without using a cube or other power. The following explains why it is preferable to use a centroid calculation using a cube or fourth power value as a method for estimating pixel shift amounts on a sub-pixel basis.

[0052] 6, 7A, 7B, and 7C show the results of simulations of pixel shift amounts performed for parabolic fitting, centroid calculation using a linear value, and centroid calculation using a cubed value. As with the above, these simulations also used two images shifted by 0.2 pixels in the horizontal direction (x-axis direction) and 0.3 pixels in the vertical direction (y-axis direction) as two images to be compared, and the results are from 500 trials using different parts of the images as template images.

[0053] FIG. 6 shows the average error and standard deviation of the pixel shift amount derived by parabolic fitting, centroid calculation using linear values, and centroid calculation using cube values.

[0054] Fig. 7A shows the distribution of x- and y-direction errors of pixel shift amounts derived by parabolic fitting. Fig. 7B shows the distribution of x- and y-direction errors of pixel shift amounts derived by centroid calculation using linear values. Fig. 7C shows the distribution of x- and y-direction errors of pixel shift amounts derived by centroid calculation using cubed values.

[0055] As shown in Fig. 7A, the x- and y-direction errors of the pixel shift amount derived by parabolic fitting vary widely. On the other hand, as shown in Fig. 7B and 7C, the x- and y-direction errors of the pixel shift amount derived by centroid calculation using linear values and centroid calculation using cubed values vary less than in the case of parabolic fitting. However, the x- and y-direction errors of the pixel shift amount derived by centroid calculation using linear values have a larger bias in the error direction than the x- and y-direction errors of the pixel shift amount derived by centroid calculation using cubed values.

[0056] 8A, 8B, 8C, 8D, 8E, and 8F show the results of a simulation in which pixel shift amounts are derived by parabolic fitting, centroid calculation using a first-power value, and centroid calculation using a third-power value, using images with different sub-pixel shift amounts in the horizontal direction (x-axis direction). 8A, 8B, 8C, 8D, 8E, and 8F show the results of the simulation performed on images of different scenes.

[0057] As shown in Figures 8A to 8F, parabolic fitting is less susceptible to pixel locking, a phenomenon in which estimation errors are biased toward integer values, than centroid calculation. However, when calculating centroids using linear values, errors tend to be biased in the same positive and negative directions for the positive and negative sub-pixel shifts in all scene images, compared to centroid calculations using cubed values. This result also shows that when calculating centroids using linear values, the error direction is more biased than when calculating centroids using cubed values.

[0058] As shown by the above simulation results, the preferred method for estimating the pixel shift amount in sub-pixel units is to calculate the center of gravity using cubed values.

[0059] 9A and 9B show the average estimation error and standard deviation in the x direction of pixel shifts calculated by centroid calculations using different powers of the normalized correlation function. As shown in FIGS. 9A and 9B, the error is smaller when calculating the pixel shifts using the third or fourth power than when calculating the pixel shifts using the first or second power. Furthermore, the error tends to increase again when using powers greater than the fourth power. Therefore, it can be seen that calculating the pixel shifts using the third or fourth power is preferable as a method for estimating pixel shifts on a subpixel basis in template matching.

[0060] As described above, the identification unit 114 identifies the positional relationship between the two images that have the greatest degree of correlation by convolving the two images to be compared in real space, and then derives the pixel shift amount in sub-pixel units by calculating the center of gravity using a cube or fourth power. This makes it possible to accurately identify the pixel shift amount between two images with little change in brightness value that are captured at a relatively short imaging interval by an area camera 130 equipped with a relatively small image sensor mounted on an artificial satellite 10 or the like.

[0061] The generation unit 116 generates transmission data for transmitting a plurality of line images and pixel shift amount information indicating the pixel shift amount to the image generation device 200 via the communication unit 150. The communication unit 150 generates transmission data in a format according to a predetermined communication method, including a plurality of line images and pixel shift amount information indicating the pixel shift amount.

[0062] The communication unit 150 transmits the transmission data to the image generation device 200 in accordance with a predetermined communication method such as the fifth generation mobile communication system (5G).

[0063] The image generating device 200 receives a plurality of line images and pixel shift amount information indicating the pixel shift amount, adjusts the horizontal and vertical positions of each of the plurality of line pixels using the pixel shift amount, and generates a concatenated image by lining up and concatenating the plurality of line images whose positions have been adjusted in the vertical direction.

[0064] For example, as shown in FIG. 10 , when the satellite 10 moves along an orbit 500, the image generation device 200 adjusts the horizontal and vertical positions of each of the plurality of line pixels using the pixel shift amount, and then vertically aligns and connects the plurality of line images whose positions have been adjusted, thereby generating a connected image as shown in FIG. 11 . This allows the image processing system 300 according to this embodiment to prevent image distortion. Because the images captured by the area camera 130 have low resolution, the processing load on the image processing device 100 when identifying the pixel shift amount by comparing the images is relatively light. Furthermore, the data transmitted to the terrestrial image generation device 200 includes, in addition to the plurality of line images, pixel shift amount information, which is a relatively small amount of data. This prevents an increase in the communication load when transmitting information from the satellite 10 to the terrestrial image generation device 200.

[0065] FIG. 12 is a flowchart showing an example of an image transmission procedure of the image processing device 100.

[0066] The acquisition unit 112 acquires a plurality of area images captured by the area camera 130 and a plurality of line images captured by the TDI camera 140 (S100). The acquisition unit 112 may acquire a plurality of area images in a first cycle, and may acquire a plurality of line images in a second cycle that is shorter than the first cycle. In other words, the number of the plurality of line images acquired by the acquisition unit 112 may be greater than the number of the plurality of area images.

[0067] The determination unit 114 determines pixel shift amounts in the horizontal and vertical directions between the multiple area images by pattern matching based on the multiple area images (S102). The determination unit 114 may, for example, derive a correlation between a first partial image and each of the multiple second partial images from a cross-correlation function obtained by convolving a first partial image in the first area image with each of the multiple second partial images in a second area image subsequent to the first area image in real space, identify the second partial image with the highest correlation, normalize the correlation between the second partial image with the highest correlation and another second partial image whose center is located in a 3×3 pixel block at the center of the second partial image with the highest correlation, derive the center of gravity of each normalized value by raising the normalized value to the third or fourth power, and determine the pixel shift amounts in the horizontal and vertical directions from the positional relationship between the center of gravity and the center of the first partial image.

[0068] The generation unit 116 generates transmission data for transmitting the plurality of line images and pixel shift amount information indicating the pixel shift amount to the image generation device 200 via the communication unit 150 (S104). The communication unit 150 transmits the transmission data to the image generation device 200 in accordance with a predetermined communication method (S106).

[0069] As described above, the data transmitted to the terrestrial image generating device 200 includes not only multiple line images but also pixel shift amount information, which has a relatively small amount of data, so that an increase in communication load when transmitting information from the artificial satellite 10 to the terrestrial image generating device 200 can be suppressed.

[0070] 13 illustrates an example of a computer 1200 in which aspects of the present invention may be embodied, in whole or in part. A program installed on the computer 1200 may cause the computer 1200 to perform operations associated with an apparatus according to an embodiment of the present invention or to function as one or more “parts” of the apparatus. Alternatively, the program may cause the computer 1200 to perform the operations or one or more “parts.” The program may cause the computer 1200 to perform a process or steps of a process according to an embodiment of the present invention. Such a program may be executed by the CPU 1212 to cause the computer 1200 to perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0071] The computer 1200 according to this embodiment includes a CPU 1212 and a RAM 1214, which are interconnected by a host controller 1210. The computer 1200 also includes a communication interface 1222 and an input / output unit, which are connected to the host controller 1210 via an input / output controller 1220. The computer 1200 also includes a ROM 1230. The CPU 1212 operates according to programs stored in the ROM 1230 and RAM 1214, thereby controlling each unit.

[0072] The communication interface 1222 communicates with other electronic devices via a network. A hard disk drive may store programs and data used by the CPU 1212 in the computer 1200. The ROM 1230 stores a boot program executed by the computer 1200 upon activation and / or programs dependent on the computer's hardware. The programs may be provided via a computer-readable recording medium such as a CD-ROM, a USB memory, or an IC card, or via a network. The programs may be installed in the RAM 1214 or the ROM 1230, which are also examples of computer-readable recording media, and executed by the CPU 1212. The information processing described in these programs is read by the computer 1200 and establishes cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be configured by implementing operations or processing of information in accordance with the use of the computer 1200.

[0073] For example, when communication is performed between computer 1200 and an external device, CPU 1212 may execute a communication program loaded into RAM 1214 and instruct communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 1212, communication interface 1222 reads transmission data stored in a transmission buffer area provided in RAM 1214 or a recording medium such as a USB memory, and transmits the read transmission data to a network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.

[0074] The CPU 1212 may also cause all or a necessary portion of a file or database stored on an external recording medium such as a USB memory to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write the processed data back to the external recording medium.

[0075] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. The CPU 1212 may perform various types of processing on data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequences of the programs, and write the results back to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. on the recording medium. For example, if multiple entries, each having an attribute value of a first attribute associated with an attribute value of a second attribute, are stored on the recording medium, the CPU 1212 may search for an entry that matches a condition specified by the attribute value of the first attribute from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0076] The above-described programs or software modules may be stored in a computer-readable storage medium on or near the computer 1200. A recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can also be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0077] A computer-readable medium may include any tangible device capable of storing instructions that are executed by an appropriate device. As a result, the computer-readable medium with instructions stored thereon comprises an article of manufacture, including instructions that can be executed to create means for performing the operations specified in the flowchart or block diagram. Examples of computer-readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable media may include floppy disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray (RTM) disc, memory stick, integrated circuit card, etc.

[0078] The computer-readable instructions may include either source code or object code written in any combination of one or more programming languages. The source code or object code may include conventional procedural programming languages. The conventional procedural programming languages may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object-oriented programming languages such as Smalltalk®, JAVA®, C++, etc., and the “C” programming language or similar programming languages. The computer-readable instructions may be provided to a processor or programmable circuitry of a programmable data processing apparatus locally or over a local area network (LAN), a wide area network (WAN) such as the Internet, etc. The processor or programmable circuitry may execute the computer-readable instructions to create means for performing the operations specified in the flowcharts or block diagrams.

[0079] Here, the computer may be a computer such as a PC (personal computer), tablet computer, smartphone, workstation, server computer, or general-purpose computer, or may be a computer system in which multiple computers are connected. Such a computer system in which multiple computers are connected is also called a distributed computing system, and is a computer in the broad sense. In a distributed computing system, the multiple computers collectively execute a program by each executing a part of the program and passing data between the computers as needed during program execution.

[0080] Examples of processors include computer processors, central processing units (CPUs), processing units, microprocessors, digital signal processors, controllers, microcontrollers, FPGAs, etc. A computer may have one or more processors. In a multiprocessor system with multiple processors, each processor executes a portion of a program and passes data between processors as needed during program execution, allowing the multiple processors to collectively execute the program. For example, in multitasking, each of the multiple processors may execute a portion of each task by switching tasks at each time slice. In this case, which portion of a program each processor executes changes dynamically. Alternatively, which portion of a program each of the multiple processors executes may be statically determined by multiprocessor-aware programming.

[0081] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

[0082] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and that the processes can be performed in any order unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, this does not mean that the processes must be performed in this order. [Explanation of symbols]

[0083] 10 satellite 100 Image processing device 110 control section 112 Acquisition Department 114 Specific section 116 Generation part 120 Storage section 130 Area Camera 140 TDI Camera 150 Communications Department 200 Image generation device 300 Image Processing System 500 orbits 1200 Computer 1210 host controller 1212 CPU 1214 RAM 1220 Input / Output Controller 1222 communication interface 1230 ROM

Claims

1. an acquisition unit that acquires a plurality of area images, each composed of M×N pixels (M and N are integers of 2 or more), which are imaged by a first imaging device mounted on a moving body and have M pixels in a first direction corresponding to the moving direction of the moving body and N pixels in a second direction intersecting the first direction, and acquires a plurality of line images, each composed of n pixels in the first direction and n×L pixels (n is an integer of 1 or more, and L is an integer of 3 or more that is greater than M and N), which are imaged by a second imaging device mounted on the moving body; a specifying unit that specifies pixel shift amounts in the first direction and the second direction between the plurality of area images based on the plurality of area images; a generation unit that generates transmission data for transmitting, via a communication unit, the plurality of line images and pixel shift amount information indicating the pixel shift amount to an image generation device that generates a connected image by connecting the plurality of line images whose positions in the first direction and the second direction are adjusted using the pixel shift amount and that are arranged in the first direction; An image processing device comprising:

2. The image processing device according to claim 1 , wherein the specifying unit specifies the pixel shift amount by comparing the plurality of area images with each other using template matching.

3. the plurality of area images include a first area image and a second area image captured subsequent to the first area image, 3. The image processing device described in claim 2, wherein the identification unit identifies the second partial image with the highest correlation among the plurality of second partial images as a target partial image by template matching a first partial image in a first region within the first area image with each of a plurality of second partial images included in a search region larger than the first region, including a second region in the second area image shifted from the first region by a first number of pixels in the first direction and a second number of pixels in the second direction, and identifies the amount of pixel shift based on the first partial image and the target partial image.

4. The image processing device described in claim 3, wherein the identification unit derives the degree of correlation between the first partial image and each of the multiple second partial images from a cross-correlation function obtained by convolving the first partial image with each of the multiple second partial images in real space.

5. The image processing device according to claim 3 , wherein the first number of pixels and the second number of pixels are determined based on a moving direction and a moving speed of the moving object and a moving direction and a moving speed of the subject relative to the first imaging device.

6. The image processing device described in claim 3, wherein the identification unit starts a search from a second partial image whose central pixel is a pixel located at the center of the second region among the plurality of second partial images, and identifies the target partial image by template matching the first partial image with second partial images in order, starting from the second partial image closest to the second partial image.

7. 4. The image processing device of claim 3, wherein the determination unit determines the pixel shift amount in sub-pixel units from the positional relationship between the center of the first partial image and the center of gravity derived from the cubed or fourth power of each normalized value of the correlation degree with the target partial image and the correlation degree with each of multiple surrounding partial images whose centers are within a predetermined pixel range from the center of the target partial image.

8. The image processing device described in claim 3, wherein the identification unit identifies the fourth partial image with the highest correlation among the plurality of fourth partial images as another target partial image by template matching a third partial image in a third area different from the first area in the first area image with each of a plurality of fourth partial images included in a search area larger than the third area, including a fourth area shifted from the third area in the second area image by the first number of pixels in the first direction and the second number of pixels in the second direction, and identifies the pixel shift amount based on the first partial image, the target partial image, the third partial image, and the other target partial image.

9. 9. The image processing device according to claim 8, wherein the determination unit determines the pixel shift amount between the first area image and the second area image based on a statistical value of the pixel shift amount based on the first partial image and the target partial image and the pixel shift amount based on the third partial image and the other target partial image.

10. the first imaging device has an area sensor that captures the plurality of area images, The image processing device according to claim 1 , wherein the second image capturing device has a line sensor that captures the plurality of line images or a time delay integration (TDI) sensor.

11. The image processing device according to claim 1 , wherein the moving object is an aircraft or an artificial satellite.

12. An image processing device according to any one of claims 1 to 11; the first imaging device; the second imaging device; the communication unit that transmits the transmission data to the image generation device; An artificial satellite equipped with

13. A satellite according to claim 12; the image generating device that receives the transmission data and generates the combined image; An image processing system comprising:

14. acquiring a plurality of area images, each composed of M×N pixels (M and N are integers of 2 or more), with M pixels in a first direction corresponding to the moving direction of the moving body and N pixels in a second direction intersecting the first direction, captured by a first imaging device mounted on the moving body; and acquiring a plurality of line images, each composed of n pixels in the first direction and n×L pixels (n is an integer of 1 or more, and L is an integer of 3 or more that is greater than M and N), captured by a second imaging device mounted on the moving body; determining pixel shift amounts in the first direction and the second direction between the plurality of area images based on the plurality of area images; generating transmission data for transmitting, via a communication unit, the plurality of line images and pixel shift amount information indicating the pixel shift amount to an image generating device that generates a connected image by adjusting positions in the first direction and the second direction using the pixel shift amount and connecting the plurality of line images arranged in the first direction; An image processing method comprising:

15. When executed by a computer, the computer an acquisition unit that acquires a plurality of area images, each composed of M×N pixels (M and N are integers of 2 or more), which are imaged by a first imaging device mounted on a moving body and have M pixels in a first direction corresponding to the moving direction of the moving body and N pixels in a second direction intersecting the first direction, and acquires a plurality of line images, each composed of n pixels in the first direction and n×L pixels (n is an integer of 1 or more, and L is an integer of 3 or more that is greater than M and N), which are imaged by a second imaging device mounted on the moving body; a specifying unit that specifies pixel shift amounts in the first direction and the second direction between the plurality of area images based on the plurality of area images; A program for causing an image generating device that generates a concatenated image by adjusting the positions in the first direction and the second direction using the pixel shift amount and concatenating the plurality of line images arranged in the first direction, to function as a generation unit that generates transmission data for transmitting the plurality of line images and pixel shift amount information indicating the pixel shift amount via a communication unit.

Images