Image processing device, image processing system, image change detection method, and program

Abstract

The objective of the present invention is to highly accurately detect a change between images obtained by capturing the same object at different timings. In the present invention, a class composition ratio estimation means estimates a second class composition ratio indicating the ratio of classes, which are the types of objects in at least a second image among a first image and the second image obtained by capturing the same object at different timings. A change detection means compares a first class composition ratio indicating the ratio of classes, which are the types of objects in the first image, with the second class composition ratio, determines the presence or absence of a change between the first image and the second image on the basis of the comparison result, and outputs the determination result.

Description

Image processing device, image processing system, image change detection method and program

[0001] The present disclosure relates to an image processing device, an image processing system, an image change detection method, and a program.

[0002] Satellite images are widely used to detect changes in the earth's surface for purposes such as understanding disaster situations and detecting illegal logging in forests. For example, Patent Document 1 proposes an image information analysis device that analyzes the difference between two images, either aerial photographs or satellite images, of the same location taken at different times. This image information analysis device extracts shading information from each of the two images. The image information analysis device then analyzes the difference between the two images based on the difference in shading information at specific points.

[0003] The image information analysis device disclosed in Patent Document 1 is based on the use of optical images. However, in recent years, the use of synthetic aperture radar (SAR) images has been increasing. Since SAR images do not reflect changes in weather or sunlight conditions, they are expected to be able to detect changes on the earth's surface more flexibly.

[0004] Japanese Patent Application Publication No. 10-214328

[0005] However, because SAR images look different from optical images, it is difficult to create correct data for technological development and evaluation, and manual work is required. Therefore, when using images such as SAR images that require manual work for change detection, there is a problem that the cost of image change detection increases.

[0006] An image processing device according to one aspect of the present disclosure includes a class composition ratio estimation means for estimating a second class composition ratio indicating the ratio of a class, which is a type of object, in at least a second image of a first and second images of the same object captured at different times; and a change detection means for comparing the first class composition ratio, which indicates the ratio of a class, which is a type of object, in the first image, with the second class composition ratio, determining whether or not there is a change between the first image and the second image based on the comparison result, and outputting the determination result.

[0007] An image processing system according to one aspect of the present disclosure comprises a storage means for storing first and second images of the same object captured at different times, and an image processing device for detecting changes between the first and second images, wherein the image processing device comprises a class composition ratio estimation means for estimating a second class composition ratio indicating the ratio of a class that is a type of object in at least the second image, of the first image and the second image acquired from the storage means, and a change detection means for comparing the first class composition ratio indicating the ratio of a class that is a type of object in the first image with the second class composition ratio, determining whether or not there is a change between the first image and the second image based on the comparison result, and outputting the determination result.

[0008] A method for detecting changes in an image that is one aspect of the present disclosure estimates a second class composition ratio indicating the proportion of classes that are types of objects in at least a second image of a first and second image of the same object captured at different times, compares the first class composition ratio indicating the proportion of classes that are types of objects in the first image with the second class composition ratio, determines whether there is a change between the first image and the second image based on the comparison result, and outputs the determination result.

[0009] A program according to one aspect of the present disclosure causes a computer to execute the following processes: a process of estimating a second class composition ratio indicating the proportion of a class, which is a type of object, in at least a second image of a first and second images of the same object captured at different times; a process of comparing the first class composition ratio, which indicates the proportion of a class, which is a type of object, in the first image, with the second class composition ratio, determining whether or not there is a change between the first image and the second image based on the comparison result, and outputting the determination result.

[0010] According to the present disclosure, it is possible to detect changes between images of the same object captured at different times with high accuracy.

[0011] 1 is a block diagram schematically showing the configuration of an image processing system according to an embodiment. FIG. 1 is a block diagram schematically showing the configuration of an image processing device according to an embodiment. FIG. 2 is a diagram showing examples of classes used in estimating class composition ratios. FIG. 3 is a flowchart of image change detection processing in an image processing device according to an embodiment. FIG. 4 is a block diagram schematically showing the configuration of an image processing device according to an embodiment. FIG. 5 is a block diagram schematically showing the configuration of an image processing device according to an embodiment. FIG. 6 is a diagram showing an example of image division. FIG. 7 is a flowchart of image change detection processing in an image processing device according to an embodiment. FIG. 8 is a block diagram schematically showing the configuration of an image processing device according to an embodiment. FIG. 9 is a diagram showing an overview of class composition ratio correction processing in an image processing device according to an embodiment. FIG. 10 is a flowchart of image change detection processing in an image processing device according to an embodiment. FIG. 11 is a block diagram schematically showing the configuration of an image processing device according to an embodiment. FIG. 12 is a diagram showing image fragment correction processing in an image processing device according to an embodiment. FIG. 13 is a flowchart of image change detection processing in an image processing device according to an embodiment. FIG. 14 is a block diagram schematically showing the configuration of an image processing device according to an embodiment. FIG. 15 is a flowchart of image change detection processing in an image processing device according to an embodiment. FIG. 16 is a block diagram schematically showing the configuration of an image processing device according to an embodiment. FIG. 1 is a diagram illustrating an example of a hardware configuration for realizing an image processing device.

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings, the same elements are designated by the same reference numerals, and redundant explanations will be omitted as necessary.

[0013] Hereinafter, when referring to one embodiment, it means that the invention can be applied to any one of the embodiments described below or a combination of two or more embodiments, and is not limited to a specific embodiment.

[0014] 1 is a block diagram showing a schematic configuration of an image processing system according to one embodiment. The image processing system 1000 in FIG. 1 includes a receiving device 1001, a processing device 1002, a storage device 1003, and an image processing device 100. An artificial satellite 1010 transmits satellite image data DAT, such as a synthetic aperture radar (SAR) image of the Earth's surface 1020, to the image processing system 1000.

[0015] The receiving device 1001 receives data DAT via an antenna and various associated devices. The receiving device 1001 transfers the received data DAT to the processing device 1002. The processing device 1002 reconstructs an image IMG, which is a satellite image, based on the data DAT. The processing device 1002 outputs the reconstructed image IMG to the storage device 1003.

[0016] The storage device 1003 stores the image IMG received from the processing device 1002. The storage device 1003 is configured to be able to store a plurality of images. For example, the storage device 1003 may store a plurality of images captured by an artificial satellite 1010 at different times of the same point on the Earth's surface 1020.

[0017] The image processing device 100 can appropriately acquire necessary images from the storage device 1003. As a result, the image processing device 100 can compare two images captured at the same location at different times and detect whether there is a change between the two images.

[0018] The image processing device 100 will now be described. Fig. 2 is a block diagram schematically showing the configuration of the image processing device according to an embodiment. The image processing device 100 shown in Fig. 2 includes a class composition ratio estimation unit 1 and a change detection unit 2.

[0019] The class composition ratio estimation unit 1 estimates class composition ratios for each of two images captured at different times at the same location. The class composition ratio indicates a breakdown of objects captured in the image to be estimated, and is information indicating the proportion of each class that indicates the type of object. For example, if the image to be subjected to change detection is an image of the ground surface, the objects can be classified into classes based on structures, land uses, etc., such as forests, building sites, parks, rice paddies, and golf courses.

[0020] 3 is a diagram showing examples of classes used to estimate class composition ratios. As shown in Fig. 3, the classes used to estimate class composition ratios are assigned to each identification number a class indicating the type of object captured in a satellite image.

[0021] The class composition ratio P is the ratio p of each of the N classes, as shown in the following formula, for example: １ ~p Ｎ where N is an integer equal to or greater than 2. In the above formula, i is a number for identifying a class, where i is an integer between 1 and N.

[0022] In this embodiment, the class composition ratio estimation unit 1 includes a trained model constructed by inputting training data including a plurality of pairs of training images and ground truth data for the class composition ratios into a model configured by a convolutional neural network (CNN) or the like and performing machine learning. This trained model configures a classifier for objects appearing in the input image. The class composition ratio estimation unit 1 can output an estimation result of the class composition ratio P of the input image by inputting an image for which a class composition ratio is to be estimated into this trained model. The trained model included in the class composition ratio estimation unit 1 is also referred to as a first trained model.

[0023] In the following, of two images captured at the same location at different times, the image captured earlier in time will be referred to as image IMG1, and the image captured later in time will be referred to as image IMG2. In this case, the image processing device 100 uses image IMG1, captured earlier in time, as a reference and detects whether image IMG2, captured later in time, has changed. Therefore, to easily distinguish between image IMG1 and image IMG2, image IMG1 will be referred to as reference image IMG1 below, as it serves as the reference for comparison between image IMG1 and image IMG2. Image IMG2 will be referred to as target image IMG2 below, as it serves as the image to be compared using reference image IMG1 as a reference. Note that reference image IMG1 will also be referred to as the first image, and target image IMG2 will also be referred to as the second image.

[0024] The temporal relationship between image IMG1 and image IMG2 is merely an example. Depending on the application, it may be desirable to detect the difference between an image captured earlier in time and an image captured later in time. In this case, the image captured later in time may be used as the reference image. Alternatively, the image captured earlier in time may be used as the target image.

[0025] The reference image IMG1 and the target image IMG2 are, for example, satellite images, and more specifically, SAR images. To facilitate distinction, the class composition ratio estimated for the reference image IMG1 is referred to as the reference class composition ratio P1. The class composition ratio estimated for the target image IMG2 is referred to as the target class composition ratio P2. The reference class composition ratio P1 is also referred to as the first class composition ratio. The target class composition ratio P2 is also referred to as the second class composition ratio.

[0026] The change detection unit 2 compares the reference class composition ratio P1 and the target class composition ratio P2 estimated by the class composition ratio estimation unit 1. Then, if there is a difference of a predetermined value or more between the reference class composition ratio P1 and the target class composition ratio P2, the change detection unit 2 determines that a change has occurred between the reference image IMG1 and the target image IMG2.

[0027] Change detection by the change detection unit 2 will now be described. The change detection unit 2 may calculate the distance between the reference class composition ratio P1 and the target class composition ratio P2, which are expressed as vectors as described above, as the difference ΔP between the two. The difference ΔP, which is the distance between the two vectors, may be calculated using various distance vector methods, such as Euclidean distance, standard Euclidean distance, and L1 distance.

[0028] The change detection unit 2 may compare the calculated difference ΔP with a predetermined reference value REF to determine whether or not a change has occurred between the reference image IMG1 and the target image IMG2. For example, if the difference ΔP is smaller than the reference value REF, the change detection unit 2 may determine that there has been no change between the reference image IMG1 and the target image IMG2. Alternatively, for example, if the difference ΔP is larger than the reference value REF, the change detection unit 2 may determine that there has been a change between the reference image IMG1 and the target image IMG2. Note that if the difference ΔP is equal to the reference value REF, the change detection unit 2 may determine that there has been no change between the reference image IMG1 and the target image IMG2, or that there has been a change, depending on the application.

[0029] Next, a description will be given of the change detection process in the image processing device 100. Fig. 4 is a flowchart of the change detection process in an image processing device according to an embodiment.

[0030] Step S11: The class composition ratio estimation unit 1 estimates the reference class composition ratio P1 of the reference image IMG1.

[0031] Step S12: The class composition ratio estimation unit 1 estimates the target class composition ratio P2 of the target image IMG2.

[0032] Step S13: The change detection unit 2 determines whether or not there is a change between the reference image IMG1 and the target image IMG2 based on the reference class composition ratio P1 and the target class composition ratio P2.

[0033] Step S14: The motion detection unit 2 outputs the determination result DET.

[0034] At this time, the change detection unit 2 may output the determination result DET to various devices (not shown). For example, the change detection unit 2 may output the determination result DET to a display device such as a display (not shown). This allows the display device to visually indicate that there has been a change between the reference image IMG1 and the target image IMG2. At this time, the display device may display one or both of the reference image IMG1 and the target image IMG2. Furthermore, the display device may indicate areas where there has been a change in the displayed image, for example, by color. Furthermore, the display device may display the change in the proportion for each class.

[0035] Furthermore, for example, the change detection unit 2 may output the determination result DET to an audio device (not shown), which can then notify by sound that a change has occurred between the reference image IMG1 and the target image IMG2.

[0036] As described above, the image processing apparatus 100 can detect whether or not there is a change between two images based on the difference in class composition ratios.

[0037] As described above, the image processing device 100 detects whether or not there is a change between two images by comparing the class composition ratios, rather than comparing the images themselves. As a result, even in the case of an image in which it is difficult to detect a change, such as an SAR image that looks significantly different from an optical image, it is possible to detect a change in the image by estimating the class composition ratios.

[0038] Furthermore, the image processing device 100 can detect changes in an image without requiring human intervention, thereby enabling the image processing device 100 to achieve low-cost and efficient change detection in an image.

[0039] Furthermore, because SAR images contain a large amount of random components such as noise, applying general change detection methods to SAR images can result in a high number of false positives. However, with the image processing device 100, even when using images with a lot of noise, such as SAR images, the effects of noise can be mitigated by replacing changes in the image with changes in the class composition ratios.

[0040] Therefore, the image processing device 100 can detect the presence or absence of a change between two images with higher accuracy than a general change detection method that compares the images themselves.

[0041] In the first embodiment, the change detection unit 2 determines whether there is a difference between two images by performing threshold determination, but this is just one example. In the present embodiment, an image processing device that applies machine learning to determine whether there is a difference between two images will be described.

[0042] Fig. 5 is a block diagram showing a schematic configuration of an image processing device according to one embodiment. In the image processing device 200 shown in Fig. 5, the motion detection unit 2 is replaced with a motion detection unit 3, as compared with the image processing device 100 according to the first embodiment. The other configuration of the image processing device 200 is the same as that of the image processing device 100.

[0043] The change detection unit 3 has a trained model M2 that is constructed by machine learning in advance on training data that includes multiple data elements, each of which consists of a pair of two class composition ratios for two images prepared for training and ground truth data information indicating whether or not there is a difference between the two images. The trained model M2 that the change detection unit 3 has is also referred to as a second trained model.

[0044] In FIG. 5, the trained model of the class composition ratio estimation unit 1 is denoted by the symbol M1 in order to distinguish it from the trained model M2 of the change detection unit 3.

[0045] The change detection unit 3 can determine whether there is a difference between the reference image IMG1 and the target image IMG2 by inputting the reference class composition ratio P1 and the target class composition ratio P2 to be determined into the trained model M2. At this time, the change detection unit 3 may output binary data indicating whether there is a change or not as an estimation result of whether there is a difference between the reference image IMG1 and the target image IMG2. Alternatively, the change detection unit 3 may output the probability of there being a change and the probability of there being a change as an estimation result.

[0046] According to this configuration, in change detection, estimation processing is performed using the class proportions, which are elements of the class composition ratios, as feature quantities. Therefore, compared to the case of change detection based on the distance between two vectors as in the first embodiment, this configuration is more advantageous in that it can determine the difference between two images more accurately and precisely.

[0047] The above-mentioned change detection can also be performed between two images of the same area extracted from two images captured at the same location at different times. In this embodiment, an image processing device is described that divides a captured image into multiple image segments and performs change detection for each of the divided image segments.

[0048] 6 is a block diagram showing a schematic configuration of an image processing device according to one embodiment. The image processing device 300 shown in FIG. 6 is different from the image processing device 100 according to the first embodiment in that an image division unit 4 and a result integration unit 5 are added.

[0049] Image segmentation unit 4 segments each of two images IMG10 and IMG20, which were captured at different times at the same location, into a plurality of image segments of a predetermined size. Images IMG10 and IMG20 are also referred to as third and fourth images, respectively.

[0050] FIG. 7 shows an example of image division. In the example of FIG. 7, the image division unit 4 divides each of images IMG10 and IMG20 into 5 x 5 = 25 image fragments. At this time, two image fragments of the same region in images IMG10 and IMG20 are used as the reference image IMG1 and target image IMG2 described above. In FIG. 7, as an example, the image fragments in the first row from the top and the third column from the left are displayed as the reference image IMG1 and target image IMG2. Note that, hereinafter, two image fragments in the same position in images IMG10 and IMG20 are also referred to as an image fragment pair.

[0051] The result integration unit 5 integrates the results of the determination of the presence or absence of change for all image fragment pairs, and outputs an integrated determination result OUT.

[0052] Next, a description will be given of the change detection process in the image processing device 300. Fig. 8 is a flowchart of the change detection process in an image processing device according to one embodiment.

[0053] Step S31: The image dividing unit 4 divides each of the pre-division images IMG10 and IMG20 into a predetermined number of image segments.

[0054] Step S32: The class composition ratio estimation unit 1 selects one image pair from the unselected image pair as the reference image IMG1 and the target image IMG2.

[0055] Steps S11 to S13 Steps S11 to S13 are the same as those in the first embodiment, so redundant explanations will be omitted.

[0056] Step S33: The motion detection unit 2 outputs the determination results DET for the selected reference image IMG1 and target image IMG2 to the result integration unit 5.

[0057] In step S34, the result integration unit 5 determines whether change detection has been completed for all image piece pairs. If the result integration unit 5 determines that change detection has not been completed for all image piece pairs, the process returns to step S32. At this time, the result integration unit 5 may, for example, output a repeat command INS1 to the class composition ratio estimation unit 1. This may cause the class composition ratio estimation unit 1 to select another image piece pair.

[0058] In step S35, if the result integration unit 5 determines that change detection has been completed for all image piece pairs, it integrates the determination results DET for all image piece pairs into an integrated determination result OUT, and outputs the integrated determination result OUT.

[0059] Next, we consider the significance of image segmentation. Generally, when satellite images of the Earth's surface are captured, the area captured by a single satellite image is relatively large. For example, it is conceivable to capture a satellite image of an area corresponding to map information at a scale of 1 / 25,000 or 1 / 50,000. To utilize the change detection results, it is necessary to compare the satellite image used for change detection with map information, but it is generally difficult to accurately align the satellite image with the map information. For example, if alignment with map information is performed near the center of the satellite image, there may be a large misalignment between the satellite image and the map information near the edges of the satellite image.

[0060] However, as in the present embodiment, by dividing one satellite image into multiple image fragments, it is possible to align each image fragment with the map information, thereby reducing the positional deviation of each image fragment with respect to the map information.

[0061] Furthermore, since the range of the image fragments is narrower than that of the image before division, the positional deviation within the image fragments can also be reduced. For example, when aligning with map information near the center of the image fragment in Figure 7, the distance from the center to the edge is 1 / 5 of that of the image before division. Therefore, it can be expected that the positional deviation within one image fragment can be reduced to 1 / 5 of that of the image before division.

[0062] Furthermore, image segmentation enables more sensitive detection of local changes. For example, in a satellite image capturing a wide area, even if a local change occurs in the satellite image, the proportion of the changed local area relative to the entire satellite image is small, and therefore the change in class composition ratio is also small. Therefore, it is conceivable that even if a change in class composition ratio is observed, the local change may not be detected. In contrast, by segmenting a satellite image into multiple image fragments, the proportion of the area in which a local change occurs relative to the entire image fragment can be increased. As a result, the change in class composition ratio within one image fragment is greater than the change in class composition ratio in the satellite image before segmentation. As a result, by observing the change in class composition ratio in the image fragment, local changes can be detected more sensitively.

[0063] As described above, according to the image processing device 300, by performing change detection using image fragments obtained by dividing a single image, it becomes possible to more easily align an image with map information.

[0064] In the third embodiment, a configuration was described in which a single image is divided into image pieces to perform change detection. In the present embodiment, an image processing device will be described that performs change detection by adding information about surrounding image pieces to the image piece to be processed.

[0065] 9 is a block diagram showing a schematic configuration of an image processing device according to one embodiment. The image processing device 400 shown in FIG. 9 further includes a correction processing unit 6 in addition to the components of the image processing device 300 according to the third embodiment.

[0066] The correction processing unit 6 corrects the reference class composition ratios P1 and target class composition ratios P2 of all image piece pairs estimated by the class composition ratio estimation unit 1 according to rules that will be explained below.

[0067] 10 is a diagram showing an overview of the class composition ratio correction process in an image processing device according to one embodiment. Here, of the 25 image fragments of image IMG_ORG shown in Fig. 10, the correction processing unit 6 corrects the class composition ratio of the central image fragment 40 in the third row from the top and third column from the left, using the class composition ratios of the eight image fragments 41 surrounding it.

[0068] It should be noted that the number of surrounding image fragments used to correct one image fragment is not limited to this, and any number of surrounding image fragments, one or more, adjacent to one image fragment may be used for correction.

[0069] In the following, the position of an image fragment will be represented by the row number y counting from the top and the column number x counting from the left. The corrected class composition ratio PC(x, y) of the image fragment in the yth row and xth column can be obtained by correcting it using the following formula: however, α is a weighting coefficient by which P(x, y) is multiplied, and is a value greater than 0 and less than 1. w(u, v) is a weighting coefficient by which the class composition ratio of each image fragment used in the calculation is multiplied, and is a value greater than 0 and less than 1.

[0070] The other configurations of the image processing device 400 are the same as those of the image processing device 300, so a duplicated description will be omitted.

[0071] Next, a description will be given of the change detection process in the image processing device 400. Fig. 11 is a flowchart of the change detection process in an image processing device according to one embodiment.

[0072] Steps S31 and S32 Steps S31 and S32 are the same as those in the third embodiment, so a duplicated description will be omitted.

[0073] Steps S11 and S12 Steps S11 and S12 are the same as those in the first embodiment, so a duplicated description will be omitted.

[0074] Step S41: The correction processing unit 6 determines whether estimation of class composition ratios has been completed for all image piece pairs. If the correction processing unit 6 determines that estimation of class composition ratios has not been completed for all image piece pairs, the process returns to step S32. At this time, the correction processing unit 6 may, for example, output a repeat command INS2 to the class composition ratio estimation unit 1. This may cause the class composition ratio estimation unit 1 to select another image piece pair.

[0075] Step S42: If the correction processing unit 6 determines that the estimation of class composition ratios has been completed for all image fragment pairs, it corrects the reference class composition ratios P1 and target class composition ratios P2 of all image fragment pairs to corrected reference class composition ratios PC1 and target class composition ratios PC2 according to the above-mentioned rule.

[0076] In step S43, the change detection unit 2 selects the corrected base class composition ratio PC1 and target class composition ratio PC2 for the unselected image pair, and determines whether or not there has been a change between the reference image IMG1 and the target image IMG2 based on the corrected base class composition ratio PC1 and target class composition ratio PC2.

[0077] Step S44: The motion detection unit 2 outputs the determination result DET to the result integration unit 5.

[0078] In step S45, the result integration unit 5 determines whether change detection has been completed for all image fragment pairs. If the result integration unit 5 determines that change detection has not been completed for all image fragment pairs, the process returns to step S43. At this time, the result integration unit 5 may, for example, output a repeat command INS3 to the change detection unit 2. This may cause the change detection unit 2 to select another image fragment pair.

[0079] In step S46, if the result integration unit 5 determines that change detection has been completed for all image fragment pairs, it integrates the determination results DET of all image fragments into an integrated determination result OUT, as in step S35. The result integration unit 5 then outputs the integrated determination result OUT.

[0080] As shown in FIG. 11, the image processing device 500 can perform change detection that takes into account information about image pieces surrounding each image piece by adding a process for correcting the image pieces.

[0081] As explained in the third embodiment, there may be a positional deviation between the image and the map information at the edge of each image piece. Therefore, if there is a class that exists only near the edge of the image piece, it is expected that the class composition ratio estimator 1 may not be able to perform correct machine learning and estimation processing.

[0082] Therefore, in order to suppress the influence of misalignment on machine learning and estimation processing, in this embodiment, the class composition information of the image fragment of interest is corrected using information on the class composition ratios of the surrounding image fragments, thereby achieving more robust change detection compared to embodiment 3.

[0083] Fifth Embodiment In this embodiment, another configuration of an image processing device will be described that performs change detection by taking into account not only the image fragment to be processed but also information on the image fragments surrounding the image fragment.

[0084] 12 is a block diagram illustrating a configuration of an image processing device according to one embodiment. The image processing device 500 illustrated in FIG. 12 further includes an image fragment correction unit 7 in addition to the components of the image processing device 300 according to the third embodiment.

[0085] The image fragment correction unit 7 performs a correction process on each of the multiple image fragments obtained by dividing the pre-division image by the image division unit 4. Specifically, the image fragment correction unit 7 replaces a selected image fragment with an image fragment consisting of pixels of the selected image fragment and pixels extracted from a predetermined range of the surrounding image fragments. In Figure 12, the set of image fragments of images IMG10 and IMG20 after correction processing by the image fragment correction unit 7 is indicated by the symbol F.

[0086] The class composition ratio estimation unit 1 acquires the image fragment pair corrected by the image fragment correction unit 7 as the reference image IMG1 and the target image IMG2.

[0087] 13 is a diagram illustrating a correction process for an image piece in an image processing device according to an embodiment, in which the central image piece 50 in the third row from the top and the third column from the left of the 25 image pieces shown in FIG.

[0088] In Figure 13, the corrected image fragment is composed of an area consisting of the pixels of the central image fragment 50 and the pixels of the eight surrounding image fragments that are included in a ring-shaped area 51 of a predetermined width that surrounds the central image fragment.

[0089] Note that the number of surrounding image fragments used to correct one image fragment is not limited to this, and pixels in the area adjacent to one image fragment may be used for correction in any number of surrounding image fragments, one or more of which are adjacent to one image fragment.

[0090] The width of the annular region 51 can be any value, but for example, if the amount of positional deviation between the image and the map information is known (e.g., the CE90 value), the amount of positional deviation may be converted into the resolution of the image data and determined. For example, if the CE90 value is 100 m and the image resolution is 5 m, the width of the annular region may be set to 100 / 5 = 20 pixels.

[0091] The class composition ratio PC(x, y) of the image fragment after correction can be converted as follows using the class composition ratio of the image fragment before correction. Here, P(x, y) is the class composition ratio of the image fragment at row y and column x before correction. Ω(x, y) is a pixel included in the annular region 51 extracted from the image fragments surrounding the image fragment at row y and column x before correction. P(Ω(x, y)) is the class composition ratio of the annular region 51 (Ω(x, y)). α is a weighting coefficient by which P(x, y) is multiplied. β is a weighting coefficient by which P(Ω(x, y)) is multiplied. Note that P ｃ To keep the magnitude of (x,y) the same as P(x,y) and P(Ω(x,y)), in equation [5], α and β are divided by α+β.

[0092] Note that equation [5] conceptually shows the conversion of class composition ratios, and the calculation of equation [5] is not performed in the class composition ratio estimation unit 1. The class composition ratio estimation unit 1 simply estimates the class composition ratios for the reference image IMG1 and target image IMG2, which are a pair of image fragments after correction, in the same manner as in the above-described embodiment.

[0093] The other configurations of the image processing device 500 are the same as those of the image processing device 300, so a duplicated description will be omitted.

[0094] Next, a description will be given of the change detection process in the image processing device 500. Fig. 14 is a flowchart of the change detection process in an image processing device according to one embodiment.

[0095] Step S31 Step S31 is the same as in the third embodiment, so a duplicated description will be omitted.

[0096] Step S51: The image fragment correcting unit 7 corrects each image fragment according to the above-mentioned rules, and updates each image fragment with the corrected image fragment.

[0097] Step S52: The class composition ratio estimation unit 1 selects one image piece pair from the unselected post-correction image piece pairs as the reference image IMG1 and the target image IMG2.

[0098] Steps S11 to S13 and S33 to S35 Steps S11 to S13 and S33 to S35 are the same as those in the first and third embodiments, so redundant explanations will be omitted.

[0099] As shown in Fig. 14, the image processing device 500 can perform change detection that takes into account information about image fragments surrounding each image fragment, simply by adding a process for correcting image fragments to the process shown in Fig. 8. This makes it possible to achieve more robust change detection, similar to the fourth embodiment.

[0100] Sixth Embodiment Change detection between images is performed by detecting whether an image to be compared has changed relative to a reference image. However, as will be explained below, it is also possible to perform change detection in a similar manner by inputting the class composition ratio of the reference image into an image processing device instead of the reference image. Below, we will explain an image processing device that performs change detection based on the class composition ratio of the reference image and the image to be compared.

[0101] 15 is a block diagram illustrating a configuration of an image processing device according to an embodiment. In the image processing device 600 of FIG. 15, unlike the image processing device 100, a reference class composition ratio P1 estimated for the reference image IMG1 is input to the motion detection unit 2 instead of the reference image IMG1.

[0102] In this case, the reference class composition ratio P1 is acquired in advance prior to the change detection process in the image processing device 600. The previously acquired reference class composition ratio P1 may be stored in any storage device, for example, the storage device 1003 in Fig. 1. The image processing device 600 may acquire the reference class composition ratio P1 from the storage device as necessary.

[0103] Next, a description will be given of the change detection process in the image processing device 600. Fig. 16 is a flowchart of the change detection process in an image processing device according to one embodiment.

[0104] Step S61: The motion detection unit 2 obtains the reference class composition ratio P1 of the reference image IMG1.

[0105] Steps S12 to S14 The subsequent steps S12 to S14 are the same as those in the first embodiment, so a duplicated description will be omitted.

[0106] As described above, according to the image processing device 600, by inputting the class composition ratio of the reference image instead of inputting the image that serves as the basis for comparison, it is possible to perform change detection in the same way as the image processing device 100.

[0107] In this case, the amount of data transmitted from the storage device holding the data to the image processing device can be reduced, which enables faster processing and a lighter transmission load. In particular, since the amount of image data is generally large, transmitting the class composition ratio instead of the image itself is expected to effectively reduce the communication load.

[0108] Embodiment 7 In the third to fifth embodiments, the configuration for integrating the change detection results between divided images was described. In this case, by visually displaying the integration determination results, it is possible to provide useful information to the user.

[0109] 17 is a block diagram showing a schematic configuration of an image processing device according to an embodiment of the present invention. Compared with the image processing device 300, the image processing device 700 in FIG. 17 further includes a display device 8.

[0110] The display device 8 is a device that displays the integrated judgment result OUT received from the result integration unit 5 so that it can be visually recognized by the user.

[0111] Next, a first example of the display on the display device 8 will be described. FIG. 18 is a diagram showing the first example of the display on the display device. As shown in FIG. 18, a division boundary may be displayed on the image IMG_ORG before division. Image pieces for which a change has been detected may be displayed by coloring, shading, or by surrounding them with a thick frame. In FIG. 18, the coloring, shading, and thick frame surroundings are indicated by reference numerals 71 to 73, respectively.

[0112] Next, a second example of the display on the display device 8 will be described. FIG. 19 is a diagram showing a second example of the display on the display device. In FIG. 19, similar to FIG. 18, a division boundary is displayed on the image IMG_ORG before division. The user can select a specific image fragment, for example, by operating an input device attached to or connected to the display device 8. Specifically, the user can select a specific image fragment, for example, by moving the mouse cursor to the desired image fragment and then clicking the mouse.

[0113] At this time, the display device 8 may display a pop-up 74 next to the selected image fragment, showing information about the image fragment. This pop-up may display text information such as the degree of change between the images, for example, the amount of change in class composition ratio.

[0114] The display device may also display the class with the highest ratio before and after the change on the selected image fragment. In FIG. 19 , the image fragment 75 is separated by a diagonal line, and two triangles 75A and 75B are displayed. The upper left triangle 75A displays, in color, the class with the highest ratio in the reference image IMG1 before the change. The lower right triangle 75B displays, in color, the class with the highest ratio in the target image IMG2 after the change. The ratios of the triangles 75A and 75B may be represented by the color intensity or transparency. For simplicity, the triangle 75A is shown in black and the triangle 75B is shown in white. One of the triangles 75A and 75B is also referred to as the first region, and the other as the second region.

[0115] This display method allows the user to easily understand whether there is a significant change between the two images, enough to change the class with the largest ratio, by comparing the colors of the two triangles. Also, by displaying the ratio as color intensity or transparency, the magnitude of the ratio can be easily and roughly grasped.

[0116] Other Embodiments The present disclosure has been described above with reference to the embodiments, but the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0117] In the above-described embodiment, the image to be subjected to change detection is, for example, a SAR image captured by a satellite, but this is merely an example. For example, the image to be subjected to change detection may be a type of satellite image other than a SAR image, such as an optical image. Furthermore, the image to be subjected to change detection is not limited to a satellite image, but may be an image captured by various imaging means other than a satellite, such as an aerial photograph captured by a manned aircraft or an aircraft known as a drone.

[0118] In the image processing devices according to the third to seventh embodiments, the change detection unit 2 may be replaced with the change detection unit 3 according to the second embodiment, and the trained model may be used to determine whether or not there is a change between two images.

[0119] In the image processing devices according to the second to fifth and seventh embodiments, the reference class composition ratio P1 may be input to the change detection unit 2 instead of the reference image IMG1 to determine whether or not there is a change between the two images.

[0120] The image processing device 700 according to the seventh embodiment has been described as a modified example of the image processing device 100 according to the first embodiment, but may have other configurations as appropriate. For example, the same display may be performed by providing a display device 8 in an image processing device according to any of the above-described embodiments other than the image processing device 100.

[0121] In the above-described embodiment, the determination unit and wavelength instruction unit according to the present disclosure have been described mainly as hardware configurations, but this is not limited thereto, and any processing can also be realized by having a CPU (Central Processing Unit) execute a computer program. In this case, the computer program can be stored using various types of non-transitory computer-readable medium and supplied to the computer. Non-transitory computer-readable medium includes various types of tangible storage medium. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, programmable ROMs (PROMs), erasable PROMs (EPROMs), flash ROMs, and random access memories (RAMs)). The program may also be supplied to a computer by various types of transient computer-readable media. Examples of transient computer-readable media include electric signals, optical signals, and electromagnetic waves. The transient computer-readable medium can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0122] An example of the hardware configuration of the determination unit and wavelength indication unit is shown below. FIG. 20 is a diagram showing an example of the hardware configuration for realizing an image processing device. The image processing device can be realized by a computer 9000 such as a dedicated computer or a personal computer (PC). However, the computer does not need to be physically single; multiple computers may be used when performing distributed processing. As shown in FIG. 10, the computer 9000 has a CPU (Central Processing Unit) 9001, a ROM (Read Only Memory) 9002, and a RAM (Random Access Memory) 9003, which are interconnected via a bus 9004. Note that although an explanation of the OS software for operating the computer will be omitted, it is assumed that the computer that constructs this network analysis system also has such software.

[0123] An input / output interface 9005 is also connected to the bus 9004. To the input / output interface 9005, for example, an input unit 9006 including a keyboard, mouse, sensor, etc., a display including a CRT, LCD, etc., an output unit 9007 including headphones, speakers, etc., a storage unit 9008 including a hard disk, etc., and a communication unit 9009 including a modem, terminal adapter, etc. are connected.

[0124] The CPU 9001 executes various processes (processing of an image processing device in this embodiment) in accordance with various programs stored in the ROM 9002 or various programs loaded from the storage unit 9008 to the RAM 9003. A graphics processing unit (GPU) may be provided to execute various processes (processing of an image processing device in this embodiment) in accordance with various programs stored in the ROM 9002 or various programs loaded from the storage unit 9008 to the RAM 9003, similar to the CPU 9001. The GPU is suitable for performing routine processing in parallel, and by applying it to neural network processing, for example, it is possible to improve processing speed compared to the CPU 9001. The RAM 9003 also stores data necessary for the CPU 9001 and the GPU to execute various processes, as appropriate.

[0125] The communication unit 9009 performs communication processing via the Internet (not shown), for example, transmits data provided by the CPU 9001, and outputs data received from a communication partner to the CPU 9001, RAM 9003, and storage unit 9008. The storage unit 9008 exchanges data with the CPU 9001 and stores and erases information. The communication unit 9009 also performs communication processing of analog or digital signals with other devices.

[0126] The input / output interface 9005 is also connected to a drive 9010 as needed, and, for example, a magnetic disk 9011, an optical disk 9012, a flexible disk 9013, or a semiconductor memory 9014 is appropriately attached, and computer programs read from these are installed in the memory unit 9008 as needed.

[0127] Each drawing is merely an example for describing one or more embodiments. Each drawing may not relate to only one particular embodiment, but may also relate to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0128] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0129] (Supplementary Note 1) An image processing device comprising: a class composition ratio estimation means for estimating a second class composition ratio indicating a ratio of a class that is a type of object in at least a second image of a first and second images of the same object captured at different times; and a change detection means for comparing the first class composition ratio indicating the ratio of a class that is a type of object in the first image with the second class composition ratio, determining whether or not there is a change between the first image and the second image based on a comparison result, and outputting the determination result.

[0130] (Supplementary Note 2) The image processing device described in Supplementary Note 1, wherein the first and second class composition ratios are vector quantities whose elements are the proportions of multiple classes, and the change detection means calculates the distance between the two vectors of the first and second class composition ratios, and determines that there is a change between the first image and the second image when the distance is greater than a predetermined value, and determines that there is no change between the first image and the second image when the distance is smaller than the predetermined value.

[0131] (Supplementary Note 3) The image processing device according to Supplementary Note 1 or 2, wherein the class composition ratio estimation means has a first trained model constructed by machine learning training data including a plurality of pairs each consisting of a training image and ground truth data of the class composition ratio of an object in the training image, and inputs an image to be estimated into the first trained model to estimate the class composition ratio of the image to be estimated.

[0132] (Supplementary Note 4) The image processing device described in any one of Supplementary Notes 1 to 3, wherein the change detection means has a second trained model constructed by machine learning training data including a plurality of data elements each consisting of two class composition ratios for training and ground truth data indicating the presence or absence of a change between two images corresponding to the two class composition ratios for training, and the first and second class composition ratios are input to the second trained model and the judgment result is output.

[0133] (Supplementary Note 5) The image processing device according to any one of Supplementary Notes 1 to 4, wherein the class composition ratio estimation means estimates the first class composition ratio of the first image.

[0134] (Supplementary Note 6) The image processing device according to Supplementary Note 5, further comprising: an image division means for dividing one image into two or more predetermined number of image fragments; and a result integration means for integrating and outputting a plurality of the judgment results into an integrated judgment result, wherein the image division means divides each of the third and fourth images into the predetermined number of image fragments, the class composition ratio estimation means selects two image fragments at the same position in the third and fourth images as the first and second images, respectively, and estimates the first and second class composition ratios, and the result integration means integrates the predetermined number of judgment results output by the change detection means for the predetermined number of pairs of two image fragments, and outputs the integrated judgment result.

[0135] (Supplementary Note 7) The image processing device according to Supplementary Note 6, further comprising a correction processing means that corrects the class composition ratio estimated by the class composition ratio estimation means for one of the image pieces based on the class composition ratios estimated by the class composition ratio estimation means for one or more image pieces adjacent to the one of the image pieces, and outputs the corrected class composition ratios, wherein the correction processing means corrects the first and second class composition ratios, and the change detection means determines whether or not there is a change between the first image and the second image based on the corrected first and second class composition ratios.

[0136] (Supplementary Note 8) The image dividing means divides the one image into a grid pattern, and the correction processing means corrects the class composition ratio of the one image fragment based on the class composition ratios estimated by the class composition ratio estimating means for eight image fragments adjacent to the one image fragment, and the class composition ratio P(x, y) of the image fragment in the y-th row counting from one end in the vertical direction and the x-th row counting from one end in the horizontal direction is corrected to a corrected class composition ratio PC(x, y) using the following formula: where α is a weighting coefficient by which P(x, y) is multiplied, and is a value greater than 0 and less than 1; w(u, v) is a weighting coefficient by which the class composition ratio of each image fragment is multiplied, and is a value greater than 0 and less than 1, and satisfies the following formula: 8. The image processing device of claim 7.

[0137] (Supplementary Note 9) The image processing device according to Supplementary Note 6, further comprising an image fragment correction means that performs a correction process to replace one of the image fragments with an image fragment consisting of pixels of the one image fragment and pixels of an area adjacent to the one image fragment in one or more image fragments adjacent to the one image fragment, and the class composition ratio estimation means selects two image fragments at the same positions of the third and fourth images after the correction process has been performed by the image fragment correction means as the first and second images, respectively.

[0138] (Appendix 10) An image processing device described in any one of Appendices 6 to 9, further comprising a display means for displaying a boundary indicating the area of ​​the predetermined number of image fragments in the first image or the second image based on the integrated judgment result, and displaying areas judged to have changed in a manner distinct from areas that have not changed.

[0139] (Supplementary Note 11) The image processing device described in Supplementary Note 10, wherein the display means displays text information indicating the amount of change between the first class composition ratio and the second class composition ratio for the area determined to have changed.

[0140] (Appendix 12) The image processing device described in Appendix 10, wherein the display means displays the class having the largest value in the first class composition ratio and the class having the largest value in the second class composition ratio for the area determined to have changed.

[0141] (Supplementary Note 13) The image processing device described in Supplementary Note 12, wherein the display means displays the class having the largest value in the first class composition ratio in a first region within the region where it is determined that the change has occurred in a first color, and displays the magnitude of the value of the class having the largest value in the first class composition ratio in a shade or transmittance of the first color, and displays the class having the largest value in the second class composition ratio in a second region within the region where it is determined that the change has occurred in a second color, and displays the magnitude of the value of the class having the largest value in the second class composition ratio in a shade or transmittance of the second color.

[0142] (Supplementary Note 14) The image processing device according to any one of Supplementary Notes 1 to 4, wherein the change detection means acquires the first class composition ratio stored in advance in an external storage means.

[0143] (Supplementary Note 15) The image processing device according to any one of Supplementary Notes 1 to 14, wherein the first and second images are satellite images captured at different times of the same point on the earth's surface.

[0144] (Supplementary Note 16) The image processing device according to Supplementary Note 15, wherein the satellite image is a synthetic aperture radar image.

[0145] (Supplementary Note 17) An image processing system comprising: a storage means for storing first and second images captured at different times of the same object; and an image processing device for detecting a change between the first image and the second image, wherein the image processing device comprises: a class composition ratio estimation means for estimating a second class composition ratio indicating a ratio of a class that is a type of object in at least the second image out of the first and second images acquired from the storage means; and a change detection means for comparing the first class composition ratio indicating the ratio of a class that is a type of object in the first image with the second class composition ratio, determining whether or not there is a change between the first image and the second image based on a comparison result, and outputting the determination result.

[0146] (Supplementary Note 18) A method for detecting a change in an image, comprising: estimating a second class composition ratio indicating a ratio of a class that is a type of object in at least a second image of a first and second images captured at different times of the same object; comparing the first class composition ratio indicating the ratio of a class that is a type of object in the first image with the second class composition ratio; determining whether or not there is a change between the first image and the second image based on the comparison result; and outputting the determination result.

[0147] (Supplementary Note 19) A program that causes a computer to execute the following processes: a process of estimating a second class composition ratio that indicates the ratio of a class that is a type of object in at least a second image of a first and second images captured at different times of the same object; a process of comparing the first class composition ratio that indicates the ratio of a class that is a type of object in the first image with the second class composition ratio, determining whether or not there is a change between the first image and the second image based on the comparison result, and outputting the determination result.

[0148] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the invention.

[0149] This application claims priority based on Japanese Patent Application No. 2023-205534, filed December 5, 2023, the disclosure of which is incorporated herein by reference in its entirety.

[0150] REFERENCE SIGNS LIST 1 Class composition ratio estimation unit 2, 3 Change detection unit 4 Image segmentation unit 5 Result integration unit 6 Correction processing unit 7 Image fragment correction unit 8 Display device 100, 200, 300, 400, 500, 600, 700 Image processing device 1000 Image processing system 1001 Receiving device 1002 Processing device 1003 Storage device 1010 Artificial satellite 1020 Earth's surface 9000 Computer 9001 CPU 9002 ROM 9003 RAM 9004 Bus 9005 Input / output interface 9006 Input unit 9007 Output unit 9008 Storage unit 9009 Communication unit 9010 Drive 9011 Magnetic disk 9012 Optical disk 9013 Flexible disk 9014 Semiconductor memory DAT Data DET Judgment results M1, M2 Trained model OUT Integrated judgment result P1 Reference class composition ratio P2 Target class composition ratio

Claims

1. A class composition ratio estimation means for estimating a second class composition ratio that shows the proportion of the class of objects in at least the second image, from first and second images of the same object taken at different times, The system includes a change detection means that compares a first class composition ratio, which indicates the proportion of the class of the object types in the first image, with a second class composition ratio, determines whether there is a change between the first image and the second image based on the comparison result, and outputs the determination result. Image processing device.

2. The first and second class composition ratios are vector quantities whose elements are the proportions of multiple classes. The aforementioned change detection means is The distance between the two vectors of the first and second class composition ratios is calculated, When the aforementioned distance is greater than a predetermined value, it is determined that there is a change between the first image and the second image. When the aforementioned distance is smaller than a predetermined value, it is determined that there is no change between the first image and the second image. The image processing apparatus according to claim 1.

3. The class composition ratio estimation means is The system has a first trained model constructed by machine learning on training data that includes multiple pairs of training images and ground truth data of the class composition ratio of objects within the training images. The image to be estimated is input into the first trained model to estimate the class composition ratio of the image to be estimated. The image processing apparatus according to claim 1 or 2.

4. The aforementioned change detection means is The system has a second trained model constructed by machine learning on training data that includes multiple data elements, each consisting of two class composition ratios for training and ground truth data indicating whether or not there is a change between two images corresponding to the two class composition ratios for training. The first and second class composition ratios are input to the second trained model, and the judgment result is output. The image processing apparatus according to claim 1 or 2.

5. The class composition ratio estimation means estimates the first class composition ratio of the first image. The image processing apparatus according to claim 1 or 2.

6. An image division means for dividing one image into two or more predetermined image segments, The system further comprises a result integration means for integrating multiple judgment results into a single integrated judgment result and outputting it, The image division means divides each of the third and fourth images into the predetermined number of image pieces. The class composition ratio estimation means selects two image fragments at the same position in the third and fourth images as the first and second images, respectively, and estimates the class composition ratios of the first and second images. The result integration means integrates the predetermined number of determination results output by the change detection means for the predetermined number of pairs of two image fragments and outputs the integrated determination result. The image processing apparatus according to claim 5.

7. The system further includes a correction processing means that corrects the class composition ratio estimated by the class composition ratio estimation means for one image piece based on the class composition ratio estimated by the class composition ratio estimation means for one or more image pieces adjacent to the one image piece, and outputs the corrected class composition ratio. The correction processing means corrects the first and second class composition ratios, The change detection means determines whether or not there is a change between the first image and the second image based on the corrected first and second class composition ratios. The image processing apparatus according to claim 6.

8. A storage means for holding first and second images of the same object captured at different times, The system includes an image processing device for detecting changes between the first image and the second image, The aforementioned image processing device is A class composition ratio estimation means for estimating a second class composition ratio that indicates the proportion of classes of objects in at least the second image, from among the first and second images acquired from the storage means, The system includes a change detection means that compares a first class composition ratio, which indicates the proportion of classes of objects in the first image, with a second class composition ratio, determines whether there is a change between the first image and the second image based on the comparison result, and outputs the determination result. Image processing system.

9. From the first and second images of the same object captured at different times, a second class composition ratio is estimated, which indicates the proportion of the class of objects in at least the second image. The system compares a first class composition ratio, which indicates the proportion of the types of objects in the first image, with a second class composition ratio, and determines whether there is a change between the first image and the second image based on the comparison result, and outputs the determination result. A method for detecting changes in an image.

10. A process for estimating a second class composition ratio, which indicates the proportion of the class of objects in at least the second image, from first and second images of the same object captured at different times, The computer is instructed to perform the following processes: compare a first class composition ratio, which indicates the proportion of the types of objects in the first image, with a second class composition ratio; determine whether there is a change between the first image and the second image based on the comparison result; and output the determination result. program.

Images