Image processing device, image processing program, and image processing method

Abstract

To provide an image processing device, image processing program and image processing method, which allow for generating a visible light image that enables accurate analysis of nighttime light without much effort.SOLUTION: An image processing device is provided, comprising a visible light image processing unit 12 configured to perform, for each local region of a visible light image of a target area captured from the sky at night, processing according to representative values of pixel values in the local region in a temperature image of the target area capture from the sky at the shooting date and time of the visible light image to generate a processed image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing program, and an image processing method. [Background technology]

[0002] Attempts are being made to observe the earth's surface from satellites and obtain a variety of information. For example, visible light images (nighttime light images) acquired at night using sensors such as the VIIRS (Visible Infrared Imager and Radiometer Suite) can be used to identify areas where nighttime light on the ground is changing, and to analyze human activity, damage in the event of a disaster, and the state of subsequent recovery. By using nighttime light images, the effects of ambient light that are often seen during the day can be avoided.

[0003] In this case, if clouds appear in the acquired nighttime light image, they block or dim the light on the ground, making it impossible to accurately grasp changes in human activity, the state of the disaster, the state of subsequent recovery, etc. For this reason, it is necessary to remove the effects of clouds. However, in the past, humans had to visually check the captured nighttime light image, select cloud-free images from previously acquired satellite images, and compare them to remove cloud areas from the nighttime light image. As a result, processing the nighttime light image required a significant amount of time.

[0004] For example, Non-Patent Document 1 describes that, in consideration of the fact that nighttime light images acquired by VIIRS sensors and the like are strongly affected by clouds, a method is described in which, for each pixel, the average value of observations on days without cloud influence over a one-year or one-month period is calculated and aggregated as observation values ​​for a one-year or one-month period, thereby creating yearly and monthly cloud-free nighttime light images.

[0005] However, in order to select a "day without clouds," visual confirmation of the nighttime light image is required. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Monitoring the recovery situation from satellites! Checking nighttime lights after the Hokkaido earthquake (https: / / sorabatake.jp / 10526 / ) Summary of the Invention [Problem to be solved by the invention]

[0007] An object of the present invention is to provide an image processing device, an image processing program, and an image processing method that can generate, without much effort, a visible light image that can accurately analyze nighttime light. [Means for solving the problem]

[0008] In order to achieve the above object, the present invention includes the following embodiments.

[0009] [1] An image processing device comprising: a visible light image processing unit that generates a processed image by performing processing for each local area of ​​a visible light image taken of a target area from the sky at night according to a representative value of pixel values ​​of the local area in a temperature image taken of the target area from the sky at the date and time the visible light image was taken.

[0010] [2] The image processing device described in [1] further comprises a memory unit that stores time-series temperature images, which are a time series of temperature images taken of the target area from above at night, and a temperature image processing unit that approximates, for each of the local areas, a time series of representative values ​​of pixel values ​​of the time-series temperature images using a first periodic function, wherein the visible light image processing unit performs processing on the pixel values ​​of the visible light image according to a result of comparing, for each of the local areas, representative values ​​of pixel values ​​of the temperature image taken at the shooting date and time of the visible light image with values ​​of the visible light image at the shooting date and time in the first periodic function, to generate the processed image.

[0011] [3] The image processing device described in [2], wherein the visible light image processing unit generates the processed image by masking a local region of the visible light image in which a representative value of the pixel values ​​of the temperature image is lower than the value of the first periodic function by a predetermined value or more.

[0012] [4] The image processing device described in [1] or [2], wherein the memory unit stores a time-series image pair, which is a time series of image pairs consisting of visible light images and temperature images taken simultaneously at night from the sky of the target area, and the visible light image processing unit generates a standard night-time light image using the time series of processed images that have been processed for each local area of ​​each visible light image included in the time-series image pair according to a representative value of the pixel values ​​of the visible light image and the temperature image that makes up the image pair.

[0013] [5] The image processing device described in [4], wherein the visible light image processing unit generates a standard nighttime light image using a time series of processed images that excludes representative values ​​of pixel values ​​of the visible light image in local regions of the visible light image where the representative value of pixel values ​​of the temperature image is lower than the value of the first periodic function by a predetermined value or more.

[0014] [6] The image processing device described in [4] or [5], wherein the visible light image processing unit generates the standard night-time light image by subtracting the standard deviation of the representative pixel values ​​from an approximate value obtained by approximating a time series of representative pixel values ​​of the processed image using a second periodic function for each local region of the processed image.

[0015] [7] The image processing device described in [4] or [5], wherein the visible light image processing unit approximates a time series of representative pixel values ​​of the processed image for each local region of the processed image using a third periodic function, sets multiple intervals in the time series, calculates a standard deviation of the representative pixel values ​​for each interval, subtracts the standard deviation from the third periodic function to calculate an interval approximation value, and approximates the interval approximation values ​​for the multiple intervals using a fourth periodic function to generate the standard night-time light image.

[0016] [8] An image processing device described in any one of [4] to [7], wherein the visible light image processing unit compares the visible light image or the processed image with the standard nighttime light image to detect changes in nighttime light in the target area.

[0017] [9] An image processing program that causes a computer to function as a visible light image processing unit that generates a processed image by performing processing on each local area of ​​a visible light image taken of a target area from the sky at night in accordance with the representative pixel value of that local area in a temperature image taken of the target area from the sky on the date and time the visible light image was taken.

[0018]

[10] An image processing method characterized by generating a processed image by performing processing on each local area of ​​a visible light image of a target area photographed from the sky at night according to a representative value of the pixel values ​​of that local area in a temperature image of the target area photographed from the sky on the date and time the visible light image was photographed. [Effects of the Invention]

[0019] According to the present invention, it is possible to provide an image processing device, an image processing program, and an image processing method that can generate, without much effort, a visible light image that can accurately analyze nighttime light by using a temperature image and a visible light image acquired from the sky at night by a sensor mounted on a satellite or the like, and processing the visible light image according to the representative value of the pixels in the temperature image. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 is a functional block diagram of an example of an image processing apparatus according to an embodiment. [Figure 2] 10A and 10B are explanatory diagrams illustrating a function approximation process of a time-series temperature image by a temperature image processing unit according to an embodiment. [Figure 3] FIG. 10 is an explanatory diagram of an example of a process for generating a cloud-processed nighttime light image according to an embodiment. [Figure 4] 3A and 3B are explanatory diagrams of a standard night light image generated by a standard night light image generating unit according to an embodiment; [Figure 5]5A and 5B are explanatory diagrams of a change detection process of the change detection unit according to the embodiment; [Figure 6] 1 is a flowchart illustrating an example of an operation of an image processing device according to an embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, modes for carrying out the present invention (hereinafter referred to as embodiments) will be described with reference to the drawings.

[0022] 1 shows a functional block diagram of an example of an image processing device 100 according to an embodiment. The image processing device 100 includes a communication unit 22, a storage unit 24, and a CPU 26, and is configured as a computer that controls the entire device and performs various calculations. The CPU 26 may include an accelerator such as a GPU in addition to the CPU.

[0023] The CPU 26 functions as a temperature image processor 10, a visible light image processor 12, a display controller 20, etc., and the visible light image processor 12 includes functions as a cloud-processed nighttime light image generator 14, a standard nighttime light image generator 16, and a change detector 18. Each of the above functions is realized, for example, by the CPU 26 and a program that controls the processing operation of the CPU 26.

[0024] The communication unit 22 is configured with an appropriate interface, and performs processing such as acquiring data such as images from an external server or the like and storing the data in the storage unit 24 .

[0025] The storage unit 24 is composed of nonvolatile memory such as a hard disk drive or solid-state drive (SSD), and stores various data and information necessary for each process performed by the image processing device 100, such as programs for operating the CPU 26. The storage unit 24 may also be a digital versatile disc (DVD), compact disc (CD), magneto-optical disc (MO), flexible disk (FD), magnetic tape, electrically erasable and rewritable read-only memory (EEPROM), flash memory, etc. The storage unit 24 preferably includes random access memory (RAM), which primarily functions as a working area for the CPU 26, and read-only memory (ROM), which stores control programs such as the BIOS and other data used by the CPU 26.

[0026] In this embodiment, the data acquired by the communication unit 22 and stored in the storage unit 24 includes a time series of visible light images (nighttime light images) of a target area captured at night by a satellite using a VIIRS sensor or the like, and a time series of temperature images of the same target area captured by a satellite using a VIIRS sensor or the like on the same date and time the visible light images were captured. That is, the communication unit 22 acquires time-series image pairs consisting of simultaneously captured visible light images and temperature images from an external server or the like, and stores them in the storage unit 24 in association with the capture date and time of each image. These time-series image pairs may be captured periodically over the past three years, for example. The time series of visible light images is also referred to as time-series visible light images, and the time series of temperature images is also referred to as time-series temperature images.

[0027] Here, the shooting date and time of a temperature image taken on the same date and time as the visible light image, or a pair of visible light and temperature images taken simultaneously, means that the shooting dates and times of the visible light image and the temperature image are the same, or the time difference between the shooting dates and times of the visible light image and the temperature image is within a range in which the cloud conditions can be considered to be the same.

[0028] The temperature image processing unit 10 reads one or more temperature images stored in the storage unit 24, calculates a representative value of the pixel values ​​of the temperature image at the shooting date and time for each pixel or each region of multiple consecutive pixels of the read temperature image, and determines the presence or absence of clouds according to the representative value. The specific determination process will be described later.

[0029] Here, a "contiguous multi-pixel region" means that all pixels included in the region are in contact with one or more pixels within the region, and includes, for example, linear regions, rectangular regions, etc., and typically includes meshes obtained by dividing an image into squares of a certain size. Hereinafter, a region consisting of one pixel or multiple contiguous pixels will also be referred to as a local region.

[0030] In addition, the "representative value of pixel value" refers to the value of a single pixel, and in the case of a multi-pixel region, refers to a predetermined type of value that represents the multi-pixel region, such as the maximum, average, or median of the pixel values ​​in the region.

[0031] The pixel values ​​of the temperature image include the temperature measured by a VIIRS sensor or the like, and the representative value of the pixel values ​​is the temperature.

[0032] In this embodiment, the temperature image processor 10 determines whether the spatial state corresponding to each local region is one of three levels: no clouds, thin clouds, or thick clouds covering the ground or sea surface. These states are referred to as "no clouds," "thin clouds," and "thick clouds." A local region or a collection of local regions determined to be cloud-free is referred to as a "no-cloud region." A local region or a collection of local regions determined to have thin clouds is referred to as a "thin cloud region." A local region or a collection of local regions determined to have thick clouds is referred to as a "thick cloud region." Furthermore, the classification of the spatial state into three levels—"no clouds," "thin clouds," and "thick clouds"—is referred to as a cloud level, and values ​​representing each cloud level are predetermined, such as 1, 2, or 3. As a result of the determination, the temperature image processor 10 generates a cloud level image in which the value representing the cloud level for each local region is set as the pixel value of the pixel corresponding to that local region.

[0033] In addition, in this embodiment, the temperature image processing unit 10 generates cloud level images for all of the time-series temperature images that make up the time-series image pair, and stores each generated cloud level image in the memory unit 24 in association with the date and time of shooting of the original temperature image.

[0034] The visible light image processing unit 12 generates a processed image in which processing for a local region of the visible light image corresponding to the local region of the temperature image is changed according to the representative value of the pixel values ​​of the temperature image.

[0035] Here, a "corresponding pixel" in the visible light image refers to a pixel that represents the same position (has the same spatial coordinates) as a pixel in the temperature image, and a "corresponding local region" in the visible light image refers to a case where the representative spatial coordinates representing the position of the local region are the same as the representative spatial coordinates representing the position of the local region in the temperature image.

[0036] Here, the pixel values ​​of the visible light image include the luminance captured by a VIIRS sensor or the like, and the representative value of the pixel values ​​is the luminance.

[0037] When a visible light image is taken from a satellite, the visible light from the nearest object is measured, so the pixel value of visible light is the brightness of visible light from the ground or sea surface on a clear day, and the brightness of visible light from the top of the clouds on a cloudy or rainy day. Therefore, in areas without clouds at night, lights on the ground or sea are photographed, but on cloudy or rainy days when the ground or sea surface is covered by clouds, the lights on the ground or sea are blocked by thick clouds and scattered by thin clouds.

[0038] In this embodiment, the visible light image processing unit 12 determines the processing to be performed on the local region by referring to the cloud level determined by the temperature image processing unit 10. That is, the visible light image processing unit 12 reads one or more visible light images stored in the storage unit 24 and cloud level images generated from the temperature images that form image pairs with each of the visible light images, and performs processing on the local region of each visible light image according to the cloud level of the local region, thereby generating a processed image from each visible light image.

[0039] For example, when the visible light image processing unit 12 functions as the cloud-processed nighttime light image generating unit 14, it generates a cloud-processed nighttime light image by setting the original visible light image in cloud-free regions, transparently combining a predetermined color in thin cloud regions, and masking the thick cloud regions with another predetermined color. Here, masking refers to the process of assigning a predetermined pixel value to the pixels in the region to be processed. The masking process fills the region to be processed with the color represented by the predetermined pixel value. Generating such a cloud-processed nighttime light image allows a user visually inspecting the displayed image to distinguish between cloud-free regions where nighttime light was captured without the influence of clouds, thin cloud regions where nighttime light may be scattered by thin clouds, and thick cloud regions where nighttime light was not captured due to thick cloud cover.

[0040] Furthermore, for example, when the visible light image processing unit 12 functions as the standard night light image generating unit 16, it generates a standard night light image representing the standard night light at a given time from processed images from multiple time points. For example, the standard night light image generating unit 16 estimates cloud-free visible light images using artificial intelligence trained to estimate visible light images of the ground surface or sea surface as they would appear in the absence of clouds for the thin cloud regions of each visible light image constituting the time-series visible light image (hereinafter referred to as cloud removal processing). For each local region, the standard night light image generating unit 16 excludes representative pixel values ​​from visible light images determined to contain thick clouds, calculates an approximation function using a time series of representative pixel values ​​consisting of representative pixel values ​​from the cloud-free visible light image and representative pixel values ​​from visible light images determined to contain thin clouds (visible light images after the cloud removal processing), and generates a time series of standard night light images whose pixel values ​​are the values ​​of the approximation function. This allows for the generation of highly accurate standard night light images that eliminate the influence of clouds.

[0041] Furthermore, for example, when the visible light image processing unit 12 functions as the change detection unit 18, it generates a detection image, which is a difference image whose pixel values ​​are obtained by subtracting the pixel values ​​of the processed image at a given time from the pixel values ​​of the standard night light image generated for that time, or a detection image obtained by processing the difference image (difference processing). This detection image is an image that represents changes in night light. Note that "processing the difference image" above refers to performing masking processing and level classification processing of the difference values, which will be described later.

[0042] In this case, the change detection unit 18 performs the cloud removal process described above in the description of the standard night light image generation unit 16 on the thin cloud region in the visible light image to generate a processed image. This processed image is called the current night light image. This enables highly accurate change detection in the thin cloud region, excluding the influence of clouds.

[0043] Furthermore, the change detection unit 18 generates a processed image (current nighttime light image) by masking local areas in the visible light image that are determined to have thick clouds, for example. In this way, it is possible to clearly show thick cloud areas where changes in nighttime light are unknown.

[0044] Furthermore, in this embodiment, the visible light image processing unit 12 generates processed images for all of the time-series visible light images that make up the time-series image pair, and stores the generated processed images in the memory unit 24 in association with the shooting dates and times of the visible light images that were the source of the processed images.

[0045] The display control unit 20 reads out cloud level images, cloud-processed nighttime light images, standard nighttime light images, detected images showing changes in nighttime light, etc. from the memory unit 24 and controls a liquid crystal display device or other appropriate display device to display them.

[0046] The temperature image processing performed by the temperature image processing unit 10 will be described with reference to FIG.

[0047] When a temperature image is captured from a satellite, the temperature of the nearest object is measured, which is the temperature of the ground or sea surface on a clear day, and the temperature of the top layer of clouds on a cloudy or rainy day. Therefore, on a cloudy or rainy day when the ground or sea surface is covered with clouds, a lower temperature is measured than the ground or sea surface on a clear, cloudless day. The temperature image processing unit 10 uses this to determine the cloud level. That is, the temperature image processing unit 10 estimates a temperature that serves as a discrimination criterion for the presence or absence of clouds from multiple past temperature images, and determines the cloud level based on that temperature. This discrimination criterion temperature is called the cloud threshold value. The cloud threshold value is estimated for each local region, and an image in which the cloud threshold value is set as the pixel value for each pixel is called a cloud threshold image.

[0048] In this embodiment, the temperature image processing unit 10 estimates the cloud threshold based on the approximate value by function approximating the time series of representative pixel values ​​of the temperature image to accommodate seasonal fluctuations in the temperature of the earth's surface and sea surface, such as spring, summer (high temperature), autumn, and winter (low temperature). Furthermore, to improve the accuracy of estimating the cloud threshold, the temperature image processing unit 10 obtains the approximate value by function approximating temperature images of the same season over multiple years.

[0049] First, the temperature image processing unit 10 reads out the time-series temperature image from the storage unit 24, and generates annual data for each local region, consisting of a pair of a representative value of the pixel values ​​of the temperature image and a date and time ignoring the year of the photographing date and time.

[0050] FIG. 2 is an explanatory diagram of the function approximation process using a periodic function by the temperature image processing unit 10. In the example of FIG. 2, the horizontal axis represents date and time, and the vertical axis represents temperature. Each point shown in the example of FIG. 2 represents a representative pixel value for a local area of ​​each temperature image over a three-year period, i.e., a temperature measurement result at a certain point within the target area, and the temperature measurement results are displayed superimposed over one year. Annual data of temperature measurement results is generated similarly for other points within the target area.

[0051] Next, the temperature image processing unit 10 calculates an approximation function for each local region by performing a regression analysis on the annual data of representative pixel values ​​of the temperature image. It is preferable to use a trigonometric function, such as a sine function (sin) or a cosine function (cos), which is a periodic function, as the approximation function to approximate the annual temperature fluctuation. This periodic function is the first periodic function of the present invention.

[0052] In this case, it is desirable to calculate the approximate function while excluding outliers. Specifically, the temperature image processing unit 10 first calculates an approximate function T using all the annual data for each local region, and calculates a sample standard deviation σ using all the annual data. T Calculate from annual data (T±2σ T ) and remove the representative values ​​(outliers) that are outside the range of the approximate function T. 2 The temperature image processor 10 calculates the coefficient of determination R for each local region and compares the amount of change with a predetermined reference value. The reference value may be, for example, three decimal places (0.001). 2 If the change in is greater than or equal to the reference value, the approximation function T is updated by approximating the annual data excluding outliers, and the sample standard deviation σ is calculated using the annual data excluding outliers. T Recalculate and update the annual data (T±2σ T ) and further exclude outliers that are outside the range of the approximate function T. The coefficient of determination R of the annual data excluding outliers is 2 The process of calculating the coefficient of determination R and comparing the amount of change with the reference value is called 2The temperature image processing unit 10 repeats the process until the change in the temperature falls below the reference value or the number of updates reaches a predetermined upper limit. T is stored in the storage unit 24.

[0053] In the example in Figure 2, the curve T shows an approximation function that uses a sine function to approximate the annual data excluding outliers.

[0054] Next, for each of the temperature images constituting the time-series temperature image, the temperature image processing unit 10 calculates a sample standard deviation σ from the value of the photographing date and time of the temperature image in the approximation function T (in the example of FIG. 2, the temperature indicated by the curve T at the date ignoring the photographing date and time). T The cloud limit value (T-α·σ T ) is calculated, where α is a constant, and is set through preliminary experiments so that the cloud limit value is close to the upper limit of the representative value when thin clouds are present. For each temperature image, the temperature image processing unit 10 generates a cloud limit image in which the cloud limit value for each local region is set to the pixel value of the pixels within that local region, and stores the cloud limit image in the memory unit 24 in association with the date and time the temperature image was taken.

[0055] Next, for each temperature image constituting the time-series temperature image, the temperature image processing unit 10 calculates the difference between the temperature image and the cloud limit image taken at the same date and time as the temperature image. In this case, the difference is calculated between local areas representing the same spatial coordinates as (representative value of pixel values ​​of the temperature image - pixel value of the cloud limit image).

[0056] The temperature image processing unit 10 then determines that local regions with cloud levels where the difference value calculated as described above is equal to or greater than a predetermined thin cloud threshold are cloud-free for each temperature image constituting the time-series temperature image. It also determines that local regions with cloud levels where the difference value is less than the thin cloud threshold and equal to or greater than a predetermined thick cloud threshold are cloud-free for each temperature image. It also determines that local regions with cloud levels where the difference value is less than the thick cloud threshold are cloud-free for each temperature image. The thin cloud threshold is greater than the thick cloud threshold, and the thin cloud threshold is set to, for example, 0. The thick cloud threshold is set, for example, through prior experiments, as the upper limit of the difference value for which cloud removal processing is effective. The temperature image processing unit 10 generates a cloud level image for each temperature image by setting the pixel values ​​of the pixels within the local region to a value corresponding to the cloud level for that local region. The cloud level image is stored in the memory unit 24 in association with the date and time of capture of the temperature image.

[0057] The cloud level determination means that a local area where the representative pixel value of the temperature image is lower than at least the sum of the pixel value of the cloud limit image and the thin cloud threshold is determined to be a cloud-covered area. In this example, since the thin cloud threshold is 0, if a temperature lower than the cloud limit value determined based on the curve T shown in Figure 2 is measured, it can be determined that clouds have occurred in the position represented by the local area where the measurement was made on that date and time.

[0058] The cloud-processed nighttime light image generation process performed by the cloud-processed nighttime light image generation section 14, which is one of the functions of the visible light image processing section 12, will be described with reference to FIG.

[0059] First, cloud-processed nighttime light image generator 14 reads the time-series visible light images and cloud-level images from storage unit 24. Next, for each visible light image that makes up the time-series visible light image, cloud-processed nighttime light image generator 14 identifies thin cloud regions and thick cloud regions by referencing the cloud-level image associated with the same capture date and time as the visible light image, and generates a cloud-processed nighttime light image by performing masking processing, such as transparently combining pixel values ​​of the thin cloud regions with red and replacing pixel values ​​of the thick cloud regions with light purple. Then, cloud-processed nighttime light image generator 14 stores each of the generated cloud-processed nighttime light images in storage unit 24, correlating them with the capture date and time of the original visible light image.

[0060] Figures 3(a), (b), and (c) illustrate an example of the process for generating a cloud-processed nighttime light image. Figure 3(a) is a cloud limit image generated by the temperature image processor 10, Figure 3(b) is the temperature image that formed the cloud limit image of Figure 3(a), and Figure 3(c) is a schematic diagram of a cloud-processed nighttime light image generated by the cloud-processed nighttime light image generator 14. The diagonally shaded area in the center right of Figure 3(c) shows the transparent composite image of a thin cloud region where the temperature is higher than the cloud limit (the pixel values ​​in Figure 3(b) are slightly lighter than those in Figure 3(a)). The diagonally shaded area in the upper left of Figure 3(c) shows the thick cloud region where the temperature is even lower than the cloud limit (the pixel values ​​in Figure 3(b) are significantly lighter than those in Figure 3(a)).

[0061] A user viewing the displayed cloud-processed nighttime light image can distinguish between cloud-free areas where nighttime light could be photographed without the influence of clouds, thin cloud areas where nighttime light may be scattered by thin clouds, and thick cloud areas where nighttime light could not be photographed due to being covered by thick clouds.

[0062] The standard night light image generation process performed by the standard night light image generation section 16, which is another function of the visible light image processing section 12, will be described with reference to FIG.

[0063] The standard night light image is generated based on a visible light image (night light image). The standard night light image represents the standard night light for each target area, and even for each local area, and serves as the basis for detecting changes in night light due to factors such as power outages and the addition or deletion of facilities. Night light (measured luminance) naturally varies depending on the location, i.e., the local area. However, since the presence or absence of seasonal fluctuations and their degree vary depending on the target area, it is desirable to approximate the time series of measured luminance over a certain period of time with a function that suits the regional characteristics of the target area, and to define a standard night light image using the value of this approximate function obtained for each local area as the pixel value. Furthermore, since there is fluctuation even in night light in the same local area (presumably due to the influence of atmospheric conditions, etc.), the standard night light image takes into account the variation in the distribution of measured luminance over a certain period of time for each local area. The standard night light image is generated using the sample standard deviation σ of the measured luminance for each local area.I The value obtained by subtracting (approximation function I-σ I ) is the pixel value of the image.

[0064] Furthermore, it is desirable to use the visible light image that is the basis for the standard night light image by excluding thick cloud areas where the night light is not properly captured, and it is also desirable to use the image after performing cloud removal processing on thin cloud areas where the night light is scattered by clouds. A visible light image that has been processed to remove thick cloud areas and has been subjected to cloud removal processing on thin cloud areas is a processed image.

[0065] First, the standard night light image generator 16 reads out the time-series visible light image and the cloud level image from the storage unit 24 .

[0066] Next, for each visible light image constituting the time-series visible light images, the standard night light image generator 16 identifies thin cloud regions by referencing cloud-level images associated with the same capture date and time as the visible light image, and performs cloud removal processing on the thin cloud regions. Specifically, the standard night light image generator 16 performs cloud removal processing by inputting the image of the thin cloud region into a trained model that has previously trained the cloud removal processing and replacing the image of the thin cloud region with the output image. The trained model can be created, for example, by preparing a large number of image pairs as training data, each consisting of a visible light image of a thin cloud region and a visible light image in which the same local region as the thin cloud region is a cloud-free region and was captured on a similar date and time. The trained model is modeled as a CNN (Convolutional Neural Network) and then using deep learning to update the parameters of the training model so that when the visible light image of the thin cloud region from the image pair in the training data is input, the output approaches the visible light image of the cloud-free region from the image pair in the training data.

[0067] Next, the standard night light image generation unit 16 generates annual data for each local region consisting of a pair of representative pixel values ​​of the processed image, where the local region is a cloudless region and a thin cloud region after cloud removal processing, and the date and time of the photograph, ignoring the year of the photographed date and time.

[0068] 4(a) and (b) show explanatory diagrams of the standard night light image generated by the standard night light image generator 16. In the example of FIG. 4, the horizontal axis represents date and time, and the vertical axis represents luminance. Each point shown in the example of FIG. 4 represents a representative pixel value for a local area of ​​each visible light image over a three-year period, i.e., a night light measurement result (measured luminance) at a certain point within the target area, and the night light measurement results are displayed overlaid over a one-year period. However, measurement results for thick cloud areas are excluded. Annual data of night light measurement results is similarly generated for other points within the target area.

[0069] Next, the standard night light image generating unit 16 calculates an approximation function for each local region by performing a regression analysis on annual data of representative pixel values ​​of the processed image. Four examples of the approximation function will be explained below.

[0070] The first example is linear approximation. For target areas where nighttime light fluctuations are small throughout the year, the approximation function can be a straight line with a slope of 0, which is the average value for a certain period. An example of this is a target area with almost no snowfall. Figure 4(a) shows an example where the approximation function is a straight line. For each local area, the standard nighttime light image generator 16 calculates an approximation line I1 for the annual data of representative pixel values ​​of the processed image, and also calculates the sample standard deviation σ of the representative pixel values ​​of the visible light image. I1 Calculate the sample standard deviation σ from the approximate line I1. I1 Approximation function (I1-σ I1 Next, the standard night light image generator 16 calculates the approximation function (I1-σ I1 ) is the approximate function (I1-σ I1 ) is set as the pixel value to generate a standard night light image, and the image is stored in the storage unit 24 in association with the photographing date and time.

[0071] On the other hand, for target areas where nighttime light fluctuates seasonally, it is preferable to use periodic trigonometric functions such as the sine function (sin) or cosine function (cos) as the approximation function. Figure 4(b) shows an example where the approximation function is a sine function. Below, we will explain the second to fourth examples of approximation functions using sine functions.

[0072] The second example is a one-year approximation using trigonometric functions. This is suitable for target areas with small seasonal variations. The standard night light image generator 16 calculates a trigonometric function I2 that approximates the annual data of the representative pixel values ​​of the processed image for each local area, and also calculates the sample standard deviation σ of the annual data of the representative pixel values ​​of the visible light image. I2 Calculate the sample standard deviation σ from the trigonometric function I2. I2 Approximation function (I2-σ I2 , not shown). The trigonometric function I2 in this second example is an example of the second periodic function in the present invention. Next, the standard night light image generator 16 calculates an approximation function (I2-σ I2 ) generates a standard night light image in which pixel values ​​are set to the date and time values ​​of the photographed dates and times of each visible light image constituting the time-series visible light image, ignoring the year, and stores the image in memory 24 in association with the photographed dates and times.

[0073] The third example is an approximation using trigonometric functions divided into intervals shorter than one year. This is suitable for target areas with large seasonal variations, such as snow glare caused by accumulated snow. The standard night light image generator 16 calculates a trigonometric function I2 that approximates the annual data of the representative pixel values ​​of the processed image for each local area, and also sets multiple intervals shorter than one year for the annual data, and calculates the sample standard deviation σ of the representative pixel values ​​of the visible light image for each interval. I3 Calculate the sample standard deviation σ from the trigonometric function I2 for each interval. I3 Approximation function (I2-σ I3) is calculated. In the example of FIG. 4(b), six intervals are set, each interval being two months apart. The trigonometric function I2 in this third example is another example of the second periodic function in the present invention. The second periodic function and the third periodic function are the same. The intervals may be set by allowing overlap, and the time length may be different for each interval. Next, the standard night light image generating unit 16 calculates the approximation function (I2-σ) of each local region. I3 ) generates a standard night light image in which pixel values ​​are set to the date and time values ​​of the photographed dates and times of each visible light image constituting the time-series visible light image, ignoring the year, and stores the image in memory 24 in association with the photographed dates and times.

[0074] The fourth example is an example in which the approximation function for each interval is approximated by a trigonometric function of an even longer interval. Like the third example, this is suitable for target areas with large bias in seasonal fluctuations. The standard night light image generation unit 16 calculates a trigonometric function I2 that approximates the annual data of the representative values ​​of pixel values ​​of the processed image for each local area, and also sets multiple intervals shorter than one year for the annual data and calculates the sample standard deviation σ of the representative values ​​of pixel values ​​of the visible light image for each interval. I3 Calculate the sample standard deviation σ from the trigonometric function I2 for each interval. I3 Approximation function (I2-σ I3 Furthermore, the standard night light image generating unit 16 calculates an approximation function (I2-σ I3 ) is approximated with one trigonometric function to calculate the approximate function I4. The trigonometric function I2 in this fourth example is an example of the third periodic function in the present invention, and the approximate function I4 is an example of the fourth periodic function in the present invention. In the example of FIG. 4(b), the approximate function (I2-σ I3 ) with a trigonometric function with a one-year cycle to calculate approximation function I4. Note that intervals may be set allowing overlap, or the time length of each interval may be different. Next, standard night light image generator 16 generates a standard night light image by setting pixel values ​​based on the approximation function I4 for each local region based on the capture date and time of each visible light image constituting the time-series visible light image, ignoring the year, and stores the image in memory 24 in association with the capture date and time.

[0075] The change detection process performed by the change detection section 18, which is yet another function of the visible light image processing section 12, will be described with reference to FIG.

[0076] First, the change detection unit 18 reads out the time-series visible light image, the cloud level image, and the standard night light image from the storage unit 24 .

[0077] Next, similar to the standard night light image generation unit 16, the change detection unit 18 identifies thin cloud areas for each visible light image that constitutes the time-series visible light image by referring to the cloud level image associated with the same shooting date and time as the visible light image, and performs cloud removal processing on the thin cloud areas.

[0078] Next, change detection unit 18 compares each of the visible light images after cloud removal processing (i.e., the processed images, hereinafter referred to as current night light images) with a standard night light image associated with the same capture date and time as the current night light image. Specifically, change detection unit 18 generates a difference image between the corresponding standard night light image and current night light image. That is, change detection unit 18 calculates a difference value (pixel value of the standard night light image - representative value of the pixel values ​​of the current night light image) for each local region, and generates a difference image in which the difference value is the representative value of the pixel values ​​for each local region.

[0079] Next, for each difference image, the change detection unit 18 identifies thick cloud areas by referring to a cloud level image associated with the same capture date and time as the current nighttime light image that is the basis of the difference image, and performs masking on the thick cloud areas to generate a detection image. Note that the change detection unit 18 may refer to the cloud level image at the stage of calculating the difference values, and perform masking on thick cloud areas without calculating the difference values.

[0080] Then, the change detection unit 18 stores each of the generated difference images in the storage unit 24 in association with the shooting date and time of the current nighttime light image that is the basis of the generated difference images.

[0081] 5(a), (b), (c), and (d) are explanatory diagrams of the change detection process of the change detection unit 18. FIG. 5(a) is the standard night light image generated by the standard night light image generation unit 16, FIG. 5(b) is the current night light image generated from a visible light image captured at a certain date and time, FIG. 5(c) is a difference image generated from the standard night light image of FIG. 5(a) and the current night light image of FIG. 5(b), and FIG. 5(d) is a cloud level image generated from a temperature image captured simultaneously with the visible light image on which the current night light image was based. Note that FIG. 5(a) is the third example of the approximation function (I2-σ) described with reference to FIG. 4(b). I3 ) is a standard night light image generated by the above method. When the difference value is positive, the brightness value of the current night light image is smaller (darker) than the standard night light image, indicating a decrease in night light. When the difference value is negative, the brightness value of the current night light image is larger (brighter) than the standard night light image, indicating an increase in night light. In the difference image shown in Figure 5(c), the brightness value of the current night light image is displayed in white, with larger brightness values. This difference image can be used to detect changes in night light and accurately grasp changes in human activity on the ground, damage during a disaster, and subsequent recovery status.

[0082] The change detection unit 18 may also perform level classification processing of the difference values ​​based on the difference image. For example, a detection image may be generated in which local areas where the current night light image has a lower luminance value than the standard night light image and local areas where the current night light image has a higher luminance value than the standard night light image are color-coded, or a detection image may be generated in which processing is performed according to the magnitude of change, such as by setting multiple thresholds for the difference value and displaying the image in a display format corresponding to the difference value. In such cases, it is also preferable to clearly indicate that there is an unknown change in the thick cloud area using a display format different from the display format corresponding to the difference value.

[0083] In the above example, the difference value is (pixel value of the standard night light image - representative pixel value of the current night light image), but the difference value may also be (representative pixel value of the current night light image - pixel value of the standard night light image). In this case, however, the interpretation of the increase or decrease in night light for positive and negative difference values ​​will be reversed.

[0084] FIG. 6 shows a flow diagram of an example of the operation of the image processing device 100 according to the embodiment.

[0085] The communication unit 22 acquires a pair of time-series images of a target area taken from the sky at night by an artificial satellite or the like, and stores each image together with the date and time of the image in the storage unit 24 (S1).

[0086] The temperature image processing unit 10 reads out the time-series temperature image stored in the storage unit 24, and approximates the annual time series of representative pixel values ​​for each local region of the read-out temperature image using a periodic function T (S2).

[0087] Next, the temperature image processing unit 10 calculates the sample standard deviation σ of the representative values ​​of the pixel values ​​for each local region of the temperature image. T Calculate the sample standard deviation σ from the value of the periodic function T at each date and time. T The value obtained by subtracting the above is set as the cloud limit value at each date and time, and the cloud limit value is set as the pixel value in a local area corresponding to the local area of ​​the temperature image, thereby generating a cloud limit image (S3).

[0088] Next, the temperature image processing unit 10 calculates the difference value as (representative value of pixel value of temperature image - pixel value of cloud limit image) for the same date and time, divides the difference value into cloud-free regions where the difference value is equal to or greater than the thin cloud threshold, thin cloud regions where the difference value is equal to or greater than the thick cloud threshold but less than the thin cloud threshold, and thick cloud regions where the difference value is less than the thick cloud threshold, and ternarizes the pixel values ​​to generate a cloud level image composed of the ternarized pixel values ​​(S4). The temperature image processing unit 10 stores the generated cloud level image in the memory unit 24 in association with the date and time of shooting of the original temperature image.

[0089] The cloud-processed nighttime light image generator 14 reads the time-series visible light images and cloud level images stored in the memory unit 24, transparently composites a predetermined color into local regions of the visible light image corresponding to thin cloud regions in the cloud level image where pixel values ​​indicating the presence of thin clouds are set, and fills in local regions of the visible light image corresponding to thick cloud regions in the cloud level image where pixel values ​​indicating the presence of clouds are set with the predetermined color, thereby generating a cloud-processed nighttime light image (S5). The cloud-processed nighttime light image generator 14 stores the generated cloud-processed nighttime light images in memory unit 24 in association with the dates and times of capture of the original visible light images.

[0090] The standard night light image generator 16 reads out the time-series visible light images, standard night light images, and cloud level images stored in the memory unit 24, and performs cloud removal processing on local regions of the visible light image corresponding to thin cloud regions in which pixel values ​​indicating the presence of thin clouds are set in the cloud level image. Furthermore, for each local region of the visible light image after cloud removal processing, the standard night light image generator 16 approximates the annual time series of pixel values ​​of the visible light image that are not pixel values ​​indicating the presence of thick clouds in the cloud level image by a periodic function I2, and sets multiple intervals in the annual time series to calculate the sample standard deviation σ for each interval. I3 Then, the standard night light image generating unit 16 calculates the approximation function (I2-σ I3 ) at the shooting date and time of the visible light image, and a standard night light image is generated as an image composed of local areas with these values ​​as pixel values, and stored in memory unit 24 in association with each shooting date and time (S6).

[0091] Change detection unit 18 reads out the time-series visible light images, standard night light images, and cloud level images stored in memory unit 24, generates a difference image between a current night light image in which thin cloud regions of the visible light image have been subjected to cloud removal processing and the standard night light image for the date and time corresponding to the current night light image, and generates a detection image in which thick cloud regions of the difference image have been masked, thereby extracting changed areas where night light has changed in the target area (S7). Change detection unit 18 associates the generated detection image with the date and time the original visible light image was captured and stores it in memory unit 24.

[0092] In response to user instructions, the display control unit 20 reads out from the memory unit 24 one or a combination of two or more of a cloud level image, a cloud-processed nighttime light image, a standard nighttime light image, and a detected image showing changes in nighttime light, and controls the display device to display it.

[0093] The program for executing each step of Fig. 6 can be stored in a recording medium, or the program can be provided via a communication means. In such a case, for example, the program described above can be considered as an invention of a "computer-readable recording medium on which a program is recorded" or an invention of a "data signal."

[0094] <Modification> Modifications of the present invention will now be described. (1) In the above embodiment, the time series of the temperature images in the time series image pair (temperature images captured simultaneously with the visible light images that serve as the basis for the standard night light image) was described as being common to the time series temperature images (temperature images that serve as the basis for the cloud limit image), but they do not have to be common. The three-year time series is also an example, and the time series may be shorter or longer than three years. For example, the time series image pair may be from the most recent three years, and the time series temperature images may be from the three years up to three years ago. Alternatively, there may be no common period, such as the time series image pair being from the most recent two years, and the time series temperature images being from the five years up to three years ago.

[0095] (2) In the above embodiment and its modified examples, an example was described in which the cloud-processed nighttime light image and the detected image are generated from an image pair included in a time-series image pair, but they do not necessarily have to be included. For example, if the time-series image pair is from the past three years up to last year, the temperature image processing unit 10 can calculate the approximation function T and the sample standard deviation σ from the time-series temperature images that make up the time-series image pair. Tto generate a cloud limit image, generate a cloud level image from the temperature images that make up the image pair at a certain point in the year, and then cloud-processed night light image generation unit 14 generates a cloud-processed night light image based on the cloud level image. Also, standard night light image generation unit 16 generates a standard night light image from the time-series visible light images that make up the time-series image pair and the cloud level image, and then generate a detected image from this standard night light image and the visible light image that makes up the image pair, so that a processed image can be generated from an image pair taken at any given date and time.

[0096] (3) In the above embodiment and its modifications, the temperature image processing unit 10 generates a cloud level image by performing differential processing between the temperature image and the cloud limit image. However, the generation of the cloud limit image (i.e., calculation of the cloud limit value) and differential processing can be omitted. In this case, for example, the temperature image processing unit 10 sets the threshold A1 representing the upper limit temperature of thin clouds as (T-α·σ T ), and the threshold A2 representing the upper limit temperature of thick clouds is set to (T-α·σ T ), and the representative pixel value for each local region of the temperature image is compared with these thresholds A1 and A2. If the representative value is A1 or greater, the local region is determined to be a cloud-free region; if the representative value is less than A1 and greater than A2, the local region is determined to be a thin cloud region; and if the representative value is less than A2, the local region is determined to be a thick cloud region, and a cloud level image is generated.

[0097] (4) In the above embodiment and its modified example (3), the temperature image processing unit 10 calculates the sample standard deviation σ of the representative pixel values ​​for each local region of the temperature image. T In this example, a cloud limit image is generated (i.e., the cloud limit value is calculated) based on the above, or a threshold A1 representing the upper limit temperature of thin clouds and a threshold A2 representing the upper limit temperature of thick clouds are set. However, if the pixel values ​​of the temperature image are stable at a constant value B, the sample standard deviation σ T Instead of (B), a constant value B common to a plurality of local regions can be used. In this case, for example, the cloud limit value is (TB), the threshold value A1 is (TB), and the threshold value A2 is a value lower than (TB).

[0098] (5) In the above embodiment and its modified examples, the temperature image processor 10 generates a cloud level image (i.e., determines the cloud level) based on the first periodic function T, and the visible light image processor 12 performs processing according to the cloud level image. However, the processing by the temperature image processor 10 can be omitted. In this case, the threshold A1 representing the upper limit temperature of thin clouds and the threshold A2 representing the upper limit temperature of thick clouds are set to predetermined constant values ​​C1 and C2, respectively. The visible light image processor 12 compares the representative pixel values ​​of local regions of the temperature image with A1 and A2 to determine the cloud level and performs processing according to the determination result. These constant values ​​C1 and C2 need to be set to relatively high values ​​through prior experiments so that local regions containing thin clouds are not determined to be cloud-free regions even due to seasonal fluctuations. Therefore, this modified example in which the temperature image processor 10 is omitted is suitable for target regions with small seasonal fluctuations.

[0099] (6) The processing performed by the visible light image processor 12 in response to the three cloud levels is not limited to the processing described in the above embodiment and its modified examples. For example, the cloud-processed nighttime light image generator 14 may use a process other than transparent compositing or masking, such as surrounding the thin cloud region and the thick cloud region with different colors, to allow the user to distinguish between the regions. Furthermore, for example, the cloud-processed nighttime light image generator 14 may perform cloud removal processing on the thin cloud region, or may perform cloud removal processing and then transparent compositing a color different from that of the thick cloud region or surrounding the thick cloud region with a color different from that of the thick cloud region. Furthermore, for example, the change detector 18 may perform subtraction processing on the thin cloud region without performing cloud removal processing, and then transparent compositing a color different from that of the thick cloud region or surrounding the thick cloud region with a color different from that of the thick cloud region.

[0100] (7) In the above embodiment and its modified examples, three cloud levels have been described as examples, but two levels are also possible. That is, the cloud levels can be either cloudless or cloudy without distinguishing between thin cloud and thick cloud regions. In this case, the temperature image processing unit 10 compares the difference value obtained by performing differential processing on the temperature image and the cloud limit image with a thin cloud threshold, and determines that local regions where the difference value is equal to or greater than the thin cloud threshold are cloudless, and determines that local regions where the difference value is less than the thin cloud threshold are cloudy. Alternatively, the temperature image processing unit 10 or the visible light image processing unit 12 compares the representative value of the pixel values ​​of the temperature image for each local region with a threshold A1, and determines that local regions where the representative value is equal to or greater than A1 are cloudless, and determines that local regions where the representative value is less than A1 are cloudy. Furthermore, for example, the cloud-processed nighttime light image generator 14 masks local regions (cloud regions) determined to contain clouds to generate a cloud-processed nighttime light image, allowing a user visually viewing the displayed image to distinguish between cloud-free regions where nighttime light was captured without cloud influence and cloud regions where nighttime light was not captured accurately due to cloud influence. For example, the standard nighttime light image generator 16 generates a standard nighttime light image by excluding representative pixel values ​​of cloud regions from a time series of visible light images, thereby enabling the generation of a highly accurate standard nighttime light image that eliminates the influence of clouds. For example, the change detector 18 performs differential processing without using representative pixel values ​​of cloud regions in the visible light image and / or masks cloud regions to generate a detection image, enabling highly accurate change detection that eliminates the influence of clouds.

[0101] (8) In the above embodiment and its modified examples, the cloud level is set to two or three levels. However, the thin cloud level can also be divided into four or more levels. In this case, the temperature image processing unit 10 uses an intermediate threshold between the thin cloud threshold and the thick cloud threshold, and in addition to the above process, compares the difference value with the intermediate threshold to generate a cloud level image with the thin cloud level divided into multiple levels. Alternatively, the temperature image processing unit 10 or the visible light image processing unit 12 uses an intermediate threshold between the thresholds A1 and A2, and in addition to the above process, compares a representative value with the intermediate threshold to divide the thin cloud level into multiple levels and determine the cloud level. Then, the visible light image processing unit 12 performs cloud removal processing on the visible light image corresponding to each of the multiple levels into which the thin cloud level is divided.

[0102] (9) In the above embodiment and its modifications, the visible light image processor 12 performs cloud removal processing on thin cloud regions. However, instead of cloud removal processing, a sharpening process may be performed by applying a sharpening filter to the thin cloud regions of the visible light image. The sharpening process can suppress the scattering of nighttime light caused by thin clouds.

[0103] (10) In the above embodiment and its modified examples, an example was shown in which images were taken from the sky using an artificial satellite. However, images may also be taken from a flying object such as an airplane or balloon, or from a sensor installed at a high altitude. [Explanation of symbols]

[0104] 10 temperature image processing unit, 12 visible light image processing unit, 14 cloud-processed nighttime light image generation unit, 16 standard nighttime light image generation unit, 18 change detection unit, 20 display control unit, 22 communication unit, 24 memory unit, 26 CPU, 100 image processing device.

Claims

1. a visible light image processing unit that processes, for each local region of a visible light image taken from the sky at night of a target area, the local region of a temperature image taken from the sky at the same time as the visible light image was taken, in accordance with a representative value of pixel values ​​of the local region to generate a processed image; a storage unit for storing a time-series temperature image, which is a time series of the temperature images; a temperature image processing unit that approximates a time series of representative values ​​of pixel values ​​of the time-series temperature image for each local region by a first periodic function; and the visible light image processing unit performs processing on the pixel values ​​of the visible light image according to a result of comparing, for each local region, a representative value of pixel values ​​of a temperature image captured at the capture date and time of the visible light image with a value of the visible light image at the capture date and time in the first periodic function, to generate the processed image.

2. 2. The image processing device according to claim 1, wherein the visible light image processing unit generates the processed image by masking a local region of the visible light image in which a representative value of pixel values ​​of the temperature image is lower than the value of the first periodic function by a predetermined value or more.

3. the storage unit stores a time-series image pair, which is a time series of image pairs consisting of visible light images and temperature images simultaneously captured from the sky at night of the target area; 2. The image processing device according to claim 1, wherein the visible light image processing unit generates a standard nighttime light image using a time series of processed images obtained by processing each of the local regions of each visible light image included in the time series image pair according to a representative value of pixel values ​​of the visible light image and the temperature image that forms the image pair.

4. 4. The image processing device according to claim 3, wherein the visible light image processing unit generates the standard nighttime light image using a time series of processed images that excludes representative values ​​of pixel values ​​of the visible light image in local regions of the visible light image where a representative value of pixel values ​​of the temperature image is lower than the value of the first periodic function by a predetermined value or more.

5. 5. The image processing device according to claim 3, wherein the visible light image processing unit generates the standard night-time light image by subtracting a standard deviation of representative pixel values ​​from an approximate value obtained by approximating a time series of representative pixel values ​​of the processed image using a second periodic function for each local region of the processed image.

6. 5. The image processing device according to claim 3, wherein the visible light image processing unit approximates a time series of representative pixel values ​​of the processed image for each local region of the processed image using a third periodic function, sets a plurality of intervals in the time series, calculates a standard deviation of the representative pixel values ​​for each interval and subtracts the standard deviation from the third periodic function to calculate an interval approximation, and approximates the interval approximation for the plurality of intervals using a fourth periodic function to generate the standard night light image.

7. The image processing device according to claim 3 , wherein the visible light image processing unit compares the visible light image or the processed image with a standard nighttime light image to detect changes in nighttime light in the target area.

8. Computer, a visible light image processing unit that performs processing for each local region of a visible light image obtained by photographing a target region from the sky at night in accordance with a representative value of pixel values ​​of the local region in a temperature image obtained by photographing the target region from the sky on the same date and time the visible light image was photographed, to generate a processed image; a storage unit for storing a time-series temperature image, which is a time series of the temperature images; a temperature image processing unit that approximates a time series of representative values ​​of pixel values ​​of the time-series temperature image for each local region by a first periodic function; It functions as the visible light image processing unit performs processing on the pixel values ​​of the visible light image according to a result of comparing, for each local region, a representative value of pixel values ​​of a temperature image captured at the capture date and time of the visible light image with a value of the visible light image at the capture date and time in the first periodic function, to generate the processed image.

9. a visible light image processing step of performing processing for each local region of a visible light image taken of a target area from above at night in accordance with a representative value of pixel values ​​of the local region in a temperature image taken of the target area from above on the same date and time as the visible light image was taken, to generate a processed image; a storage step of storing a time-series temperature image, which is a time series of the temperature images; a temperature image processing step of approximating a time series of representative values ​​of pixel values ​​of the time-series temperature image for each local region by a first periodic function; and the visible light image processing step is an image processing method characterized in that, for each local region, a representative value of pixel values ​​of a temperature image captured at the date and time the visible light image was captured is compared with the value of the visible light image at the date and time of capture in the first periodic function, and processing is performed on the pixel values ​​of the visible light image according to the result to generate the processed image.

Images