Image processing device and image processing method

Abstract

An image processing device 60 comprises: a change region detection means 61 that detects, for an object in a first image, a change region indicating a change in the object in a second image with respect to the object in the first image, the second image being obtained by observing the same observation region as the observation region of the first image; an image correction means 62 that performs correction processing of reducing the change region of the second image relative to a non-change region indicating no change in the object; and an image pair output means 63 that outputs an image pair of the first image and the second image processed by the image correction means 62.

Description

[Technical field]

[0001] The present invention relates to an image processing device and an image processing method for associating an image with an image captured in the same area as the captured area of ​​the image. [Background technology]

[0002] There are change detection technologies that detect areas where the condition of the earth's surface has changed based on images taken from high altitudes, such as those taken by satellites, in order to grasp the status of damage caused by disasters such as floods, forest fires, volcanic eruptions, earthquakes, tsunamis, and droughts, the status of urban development, and the movement and retention of cargo and people.

[0003] Synthetic Aperture Radar (SAR) technology is a technology in which flying objects such as artificial satellites and aircraft transmit and receive electromagnetic waves while moving, and obtain images (hereinafter also referred to as SAR images) equivalent to images obtained by antennas with large apertures. Synthetic aperture radar is used, for example, to process the signals of waves reflected from the earth's surface and analyze surface displacement.

[0004] Hereinafter, images taken by satellites etc. will be referred to as observed images. Unless otherwise specified, observed images include optical images and SAR images.

[0005] Generally, change detection involves comparing two images obtained by observing the same area at different times. By comparing the two images, changes in one or more objects (targets) within the area are detected. In addition to ground surface displacement, detectable changes in targets include, for example, the appearance of a new target or the disappearance of a target. Hereinafter, each of the two images will be referred to as a target presence image or target map, and the two images will sometimes be referred to as an image pair. Also, an image that can identify the differences between the two images based on the results of comparing the two images will sometimes be referred to as a change map.

[0006] By superimposing two SAR images obtained by observing the same area on the ground at different times, a composite image is generated in which changes in objects within the area can be visually recognized. In other words, the composite image is equivalent to a visualized image of the photographed area. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] P. Isola et al., "Image-to-image translation with conditional adversarial networks", in Proc. Conference on Computer Vision and Pattern Recognition, 2017 [Non-Patent Document 2] M. Schmitt et al., "The SEN1-2 dataset for deep learning in SAR-optical data fusion", ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. IV-1, 2018, pp. 141-146 Summary of the Invention [Problem to be solved by the invention]

[0008] It is preferable to analyze surface displacements and the like frequently. By collecting multiple images of the same area, the frequency of analysis can be increased.

[0009] In order to collect a large number of images, it is possible to use SAR images and optical images in combination. However, in general, the flying object that captures the SAR images is a different flying object from the flying object that captures the optical images. As a result, it is not easy to collect a large number of SAR images and a large number of optical images that observe the same area. Even if SAR images and optical images that observe the same area are obtained, the way objects are captured in those images is different. Therefore, even if the SAR image and the optical image are superimposed, a preferable composite image cannot be obtained. This is because the reliability of the composite image is low when the state of the object captured in the SAR image does not match the state of the object captured in the optical image. In other words, it is hindered from collecting a large number of images suitable for generating a composite image.

[0010] Attempts have been made to convert SAR images into optical images using machine learning. For example, Non-Patent Document 1 discloses converting SAR images into optical images by an image converter using pix2pix. When training an image converter using pix2pix, a dataset of images is required as learning data. Non-Patent Document 2 discloses the use of the Sen1-2 dataset created from image data of the European Space Agency's Sentinel-1 (SAR satellite) and Sentinel-2 (optical satellite). From the dataset, images taken in the same season and in the same area are selected. For the optical images, images without clouds or shadows are selected. Then, the optical image and the SAR image converted into the optical image are superimposed.

[0011] However, when superimposing an optical image and a SAR image converted into an optical image, it is necessary to train an image converter and select an appropriate image. In other words, it takes time to obtain a composite image. In addition, if the state of an object, etc., shown in a SAR image (a SAR image converted into an optical image) does not match the state of an object, etc., shown in an optical image, a preferable composite image cannot be obtained even if they are superimposed. In other words, it is hindered from collecting a large number of images suitable for generating a composite image.

[0012] An object of the present invention is to provide an image processing apparatus and an image processing method capable of collecting a large number of images taken of the same area that can be used for analyzing surface displacements and the like. [Means for solving the problem]

[0013] The image processing device according to the present invention includes a changed area detection means for detecting a changed area in a second image obtained by observing an observation area of ​​an object in a first image that is the same as the observation area in the first image, where the object has changed, an image correction means for performing a correction process to make the changed area in the second image relatively smaller than a non-changed area in which the object has not changed, and an image pair output means for outputting an image pair of the first image and the second image processed by the image correction means. The change area detection means includes an image transformation means for transforming an object existing area in two object existing images, which are images obtained from the two observation images and in which one or more objects exist, based on the observation angle of each of the two observation images and the size of the object appearing in each of the two observation images, to generate two transformed images, and an image generation means for synthesizing the two transformed images to generate a synthesized image, determining a change in the object between the two object existing images using the synthesized image, and generating an image in which the determined change can be identified. nothing.

[0015] The image processing method according to the present invention comprises the steps of: The computer A changed area in which the object in a first image has changed is detected in a second image obtained by observing the same observation area as the observation area in the first image, and a correction process is performed to make the changed area in the second image smaller relative to a non-changed area in which the object has not changed, and an image pair of the corrected second image and the first image is output. and when detecting a changed region in the second image, the method deforms an object presence region in two object presence images, which are images obtained from each of the two observation images and in which one or more objects are present, based on the observation angle of each of the two observation images and the size of the object appearing in each of the two observation images, to generate two deformed images, synthesizes the two deformed images to generate a composite image, determines a change in the object between the two object presence images using the composite image, and generates an image in which the determined change can be identified. . Effect of the Invention

[0019] The present invention allows for the collection of multiple images of the same area that can be used to analyze ground displacements and the like. [Brief description of the drawings]

[0020] [Figure 1] 1 is a block diagram showing an example of the configuration of an image processing device according to a first embodiment. [Diagram 2] 4 is a block diagram showing an example of the configuration of an image pair determination unit in the first embodiment. FIG. [Diagram 3] FIG. 11 is an explanatory diagram showing an example of a method for generating a change map. [Figure 4] 4 is an explanatory diagram for explaining an incident angle and a range azimuth angle of an electromagnetic wave. FIG. [Diagram 5] FIG. 11 is an explanatory diagram for explaining the amount of collapse. [Figure 6] FIG. 11 is an explanatory diagram for explaining an expansion process. [Figure 7] FIG. 11 is an explanatory diagram for explaining a change in an object. [Figure 8] FIG. 11 is an explanatory diagram for explaining a noise removal process. [Figure 9] 13 is a block diagram showing an example of the configuration of a change map generating unit. FIG. [Figure 10] 4 is a flowchart showing the operation of the image processing device according to the first embodiment. [Figure 11] 5 is a flowchart showing an operation of a change map generating unit in the first embodiment. [Figure 12] 13 is a block diagram showing an example of the configuration of an image pair determination unit in an image processing device according to a second embodiment. FIG. [Figure 13] FIG. 11 is an explanatory diagram for explaining a mask process. [Figure 14] 10 is a flowchart showing the operation of an image processing device according to a second embodiment. [Figure 15] FIG. 13 is a block diagram illustrating an example of the configuration of an image pair determination unit in an image processing device according to a third embodiment. [Figure 16] FIG. 2 is an explanatory diagram for explaining a method of generating one image in an image pair. [Figure 17] 13 is a flowchart showing the operation of an image processing device according to the third embodiment. [Figure 18] FIG. 1 is a block diagram showing an example of the configuration of an information processing device capable of implementing the functions of an image processing device. [Figure 19] 1 is a block diagram showing a main part of an image processing device; [Figure 20] FIG. 13 is a block diagram showing a main part of an image processing device according to another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0022] Embodiment 1. 1 is a block diagram showing an example of the configuration of an image processing device according to the first embodiment. The image processing device 1 shown in FIG.

[0023] A set of observed images is input to the object map creation unit 10. From each of the observed images, the object map creation unit 10 extracts an image (object present image) including an object presence region where an object that is the subject of change detection exists. That is, the object map creation unit 10 creates a set of object maps (object presence images). Note that the object map creation unit 10 extracts, for example, a predetermined region from the observed image as the object map. However, it is also possible to manually extract a region from the observed image.

[0024] The object map creation unit 10 may create an object presence image in which an object exists by using other methods. For example, the object map creation unit 10 may input the object presence image to a neural network that has been trained in advance, and output the image obtained as an output as an object map in which the object can be grasped.

[0025] The set of observation images input to the object map creation unit 10 is selected, for example, from a group of images stored in advance in an image database. When the set of observation images is selected, observation images of the same area are selected. It is preferable that the two selected observation images are images taken in the same season. In addition, when the set of observation images includes an optical image, it is preferable to select an image that does not include clouds or shadows.

[0026] The image pair determination unit 20 receives the observation angles (azimuth and incidence angles) of each observed image and the size (height and width) of the object. The size of the object is determined in advance according to the object that is the target of change detection. The image pair determination unit 20 determines the degree of match (similarity) of two object maps, i.e., image pairs. Specifically, the image pair determination unit 20 determines whether an image pair is an image pair that can be used for change detection or the like. Note that an image pair that can be used for change detection or the like is two images in which the object depicted can be considered to be the same. Such an image pair is sometimes called an object-matching image pair.

[0027] Fig. 2 is a block diagram showing an example of the configuration of an image pair determination unit. The image pair determination unit 20A shown in Fig. 2 includes a change map generation unit 21, a coincidence calculation unit 22, and a coincidence determination unit 23. The image pair determination unit 20A shown in Fig. 2 corresponds to the image pair determination unit 20 shown in Fig. 1.

[0028] The change map generating unit 21 deforms each object map based on the observation angle of each observed image and the size of the object. Furthermore, the image pair determining unit 20A generates an image showing the area where the object has changed between the two object maps, that is, an object change map (hereinafter referred to as a change map), by synthesizing the deformed object maps to generate a superimposed image.

[0029] The matching degree calculation unit 22 uses the change map to calculate the matching degree (similarity) between the two object maps, i.e., the image pair. The matching degree determination unit 23 determines whether or not the image pair input to the image processing device 1 is an object-matching image pair by determining the calculated matching degree. The matching degree determination unit 23 outputs the determination result as to whether or not the image pair is an object-matching image pair. For example, when the determination result of the matching degree determination unit 23 is positive (when the matching degree determination unit 23 determines that the input image pair is an object-matching image pair), the matching degree determination unit 23 outputs the image pair as an object-matching image pair. The object-matching image pair is stored, for example, in a predetermined image database.

[0030] Fig. 3 is an explanatory diagram showing an example of a method for generating a change map. The left side of the upper part of Fig. 3 shows a state in which an artificial satellite 100 passing through orbit A captures an image of an area including a parking lot 120 at time t1. The left side of the lower part of Fig. 2 shows a state in which an artificial satellite 100 passing through orbit B, which is different from orbit A, captures an image of an area including a parking lot 120 at time t2, which is different from time t1. Time t2 is a time later than time t1.

[0031] The center of the upper part of FIG. 3 shows an example of image A (first object map 111) obtained from an observation image at time t1. Three automobiles 91, 92, and 93 exist in image A. The center of the lower part of FIG. 2 shows an example of image B (second object map 121) obtained from an observation image at time t2. Two automobiles 93 and 94 exist in image B. At time t2, automobiles 91 and 92 that existed at time t1 have disappeared. Furthermore, a new automobile 94 has appeared at time t2. That is, a new automobile 94 has appeared between time t1 and time t2.

[0032] In this example, the first object map 111 and the second object map 121 correspond to an image of a parking lot 120.

[0033] The change map generating unit 21 generates a change map 150 using images A and B. In the change map 150, an ellipse surrounded by a solid line indicates an area in which an automobile 93 that did not change from time t1 to time t2 exists. In other words, it indicates an area where there is no change. A black ellipse indicates an area in which a newly appeared automobile 94 exists. An ellipse surrounded by a dashed line indicates an area in which the vanished automobiles 91, 92 exist. In other words, the black ellipse and the ellipse surrounded by a dashed line indicate changed areas.

[0034] In addition, in the change map 150, the changed area and the unchanged area may be distinguished by an expression different from the expression exemplified in Fig. 3. As one example, the changed area and the unchanged area may be distinguished by a difference in color.

[0035] 4 is an explanatory diagram for explaining the incidence angle and azimuth angle (range azimuth angle) of an electromagnetic wave. FIG. 4 shows a first observation image 101 obtained on an orbit A and a second observation image 102 obtained on an orbit B different from the orbit A. The incidence angle θ A ,θ B corresponds to the angle from the zenith direction to the direction of the artificial satellite 100. A ,α B corresponds to the angle of the range direction relative to a reference direction (e.g., north).

[0036] Figure 5 shows the distance of the collapse (amount of collapse) l A FIG. 1 is an explanatory diagram for explaining the above. The height of the object (in this example, a car) is h, and the incidence angle of the electromagnetic wave is θ A Then, the amount of collapse l A is expressed by the following equation (1).

[0037] l A =h / tan θ A (1)

[0038] In addition, when the observed image is an optical image, the incident angle of sunlight is θ A Then, the amount of collapse l A is expressed by the following equation (2).

[0039] l A =h tan θ A (2)

[0040] In this embodiment, the case where an SAR image is used is taken as an example, so the amount of inclination for image A is hereinafter referred to as l A Let the amount of inclination for image B be l B (l B =h / tan θ B ) In addition, when an optical image is used, the amount of inclination for image B is l B (l B =h tan θ B ).

[0041] FIG. 6 is an explanatory diagram for explaining the expansion process executed by the change map generating unit 21. As shown in FIG.

[0042] As shown in FIG. 5, the change map generating unit 21 performs an expansion process on the image A (first object map 111) based on the first observation image 101 (see FIG. 4) obtained on the trajectory A. In this embodiment, the change map generating unit 21 expands the object (in this example, a car) appearing in the image A in the collapse direction of the corresponding object in the image B by a length equivalent to the collapse amount of the object in the image B. As a result, the image A after the expansion process (the first object map 112 in which the object is expanded) is obtained. In addition, the change map generating unit 21 performs an expansion process on the image B (the second object map 121) based on the second observation image 102 (see FIG. 4) obtained on the trajectory B. In this embodiment, the object appearing in the image B is expanded in the collapse direction of the corresponding object in the image A by a length equivalent to the collapse amount of the object in the image A. As a result, the image B after the expansion process (the second object map 122 in which the object is expanded) is obtained.

[0043] It should be noted that in the first object map 112 and the second object map 121 shown in FIG. 6, black regions indicate expanded regions, ie expanded regions.

[0044] FIG. 7 is an explanatory diagram for explaining a change in an object.

[0045] The change map generating unit 21 superimposes the post-expansion image A, i.e., the first object map 112, and the post-expansion image B, i.e., the second object map 122. Fig. 7 shows a schematic representation of the superimposed image (change map) 140.

[0046] It is assumed that the observation image on which image B is based was obtained later in time than the observation image on which image A is based.

[0047] In FIG. 7, region [F,B] indicates a region where an object exists in the first object map 112 but does not exist in the second object map 122. That is, region [F,B] indicates a region where an object has disappeared. Region [F,F] indicates a region where an object exists in the first object map 112 and exists in the second object map 122. That is, region [F,F] indicates a region where no change has occurred. Region [B,F] indicates a region where an object did not exist in the first object map 112 but exists in the second object map 122. That is, region [B,F] indicates a region where a new object has appeared. Region [B,B] indicates a region where an object does not exist in either the first object map 112 or the second object map 122. That is, region [B,B] indicates a region where no change has occurred.

[0048] The change map generating unit 21 generates the change map 140 based on the idea illustrated in Fig. 7. Specifically, the change map generating unit 21 generates the change map 140 that is capable of distinguishing between a changed area (an area where an object has disappeared or appeared) and a non-changed area.

[0049] FIG. 8 is an explanatory diagram for explaining the noise removal processing.

[0050] In the change map 140 and the noise-removed change map 150 shown in FIG. 8, the black area corresponds to the area [B,F] illustrated in FIG. 7. That is, the black area indicates the area where the object has disappeared. The area surrounded by the dashed line corresponds to the area [F,B] illustrated in FIG. 7. That is, the area surrounded by the dashed line indicates the area where the object has disappeared. The area surrounded by the solid line corresponds to the area [F,F] or the area [B,B] illustrated in FIG. 7. That is, the area surrounded by the solid line indicates the area where no change has occurred.

[0051] The change map generating unit 21 performs a noise removal process on the change map 140. The noise removal process is a process in which an area that is smaller than the object is regarded as noise and removed. In the example shown in FIG. 8, the change map generating unit 21 performs an opening process on the change map 140. The opening process is a process in which contraction and expansion are combined. When performing the contraction process in the opening process, the change map generating unit 21 contracts the object by a number of pixels according to the size of the object.

[0052] In this embodiment, the change map generating unit 21 outputs the change map 150 from which noise has been removed, but may output the change map 140 before noise removal processing is performed.

[0053] Fig. 9 is a block diagram showing a specific configuration example of the change map generation unit 21. The change map generation unit 21 shown in Fig. 9 includes a first collapse parameter calculation unit 211, a second collapse parameter calculation unit 212, a first expansion unit 311, a second expansion unit 312, a map generation unit 411, and a noise removal unit 511.

[0054] The first collapse parameter calculation unit 211 receives the range azimuth angle and incident angle related to image A (first object map 111) and the height of the object. The first collapse parameter calculation unit 211 calculates the amount of collapse of the object in image A using the incident angle and the height of the object. The first collapse parameter calculation unit 211 also uses the range azimuth angle to determine the collapse direction of the object in image A. The collapse direction is determined by the range azimuth angle α A The first collapse parameter calculation unit 211 outputs the first collapse parameter to the second expansion unit 312. The first collapse parameter includes at least data indicating the collapse amount of the object and data indicating the collapse direction of the object.

[0055] The second collapse parameter calculation unit 212 receives the range azimuth angle and incident angle related to image B (second object map 121) and the height of the object. The second collapse parameter calculation unit 212 calculates the amount of collapse of the object in image B using the incident angle and the height of the object. The second collapse parameter calculation unit 212 also uses the range azimuth angle to determine the collapse direction of the object in image B. The collapse direction is determined by the range azimuth angle α B The second collapse parameter calculation unit 212 outputs the second collapse parameter to the first expansion unit 311. The second collapse parameter includes at least data indicating the collapse amount of the object and data indicating the collapse direction of the object.

[0056] In addition, when an optical image is used as the observation image, the first tilt parameter calculation unit 211 calculates the range azimuth angle α A +180 degrees (or range azimuth angle α A The second tilt parameter calculation unit 212 calculates the direction indicated by the range azimuth angle α B +180 degrees (or range azimuth angle α B The direction indicated by the second tilt parameter (-180 degrees) is calculated as the tilt direction in the second tilt parameter.

[0057] Image A and the second collapse parameters are input to first dilation unit 311. First dilation unit 311 dilates the object in image A using the second collapse parameters to generate image A in which the object is dilated (first object map 112). First dilation unit 311 outputs first object map 112 to map generation unit 411.

[0058] Image B and the first collapse parameters are input to second expansion unit 312. Second expansion unit 312 expands the object in image B using the first collapse parameters to generate image B in which the object is expanded (second object map 122). Second expansion unit 312 outputs second object map 122 to map generation unit 411.

[0059] The map generating unit 411 overlays the first object map 112 and the second object map 122. That is, the map generating unit 411 synthesizes the first object map 112 and the second object map 122. Then, the map generating unit 411 determines a change (disappearance or appearance) between an object in the first object map 112 and an object in the second object map 122 corresponding to the object. The map generating unit 411 converts the overlaid image in which the first object map 112 and the second object map 122 are overlaid into an image in which changed and unchanged areas can be distinguished, and outputs the image to the noise removing unit 511 as the change map 140.

[0060] The noise removal unit 511 performs an opening process on the change map 140 and outputs the image from which noise has been removed as a correct change map.

[0061] Next, the operation of the image processing apparatus will be described with reference to the flowchart of FIG.

[0062] The image processing device 1 receives pairs of observed images (step S11). In the image processing device 1, the object map creation unit 10 creates pairs of object maps from each of the observed images (step S12). The change map generation unit 21 in the image pair determination unit 20A generates a change map from the pairs of object maps (step S13).

[0063] FIG. 11 is a flowchart showing the operation of the change map generating section 21, that is, the process of step S13.

[0064] A pair of object maps is input to the change map generating unit 21. The pair of object maps corresponds to the first object map 111 and the second object map 121 shown in FIG.

[0065] Meta information of one observation image is input to the first collapse parameter calculation unit 211. Meta information of the other observation image is input to the second collapse parameter calculation unit 212. In general, available observation images are accompanied by meta information (metadata) such as the shooting time, shooting point (e.g., latitude and longitude of the center of the observation image), and electromagnetic wave irradiation direction (observation orientation). The first collapse parameter calculation unit 211 calculates a range azimuth angle α A and the incidence angle θ A The second collapse parameter calculation unit 212 extracts the range azimuth angle α B and the incidence angle θ B (Step S131).

[0066] It is not essential that the first collapse parameter calculation unit 211 and the second collapse parameter calculation unit 212 extract the range azimuth angle and the incidence angle from the meta information. For example, a means other than the first collapse parameter calculation unit 211 and the second collapse parameter calculation unit 212 may extract the range azimuth angle and the incidence angle from the meta information. In that case, the means supplies the extracted range azimuth angle and incidence angle to the first collapse parameter calculation unit 211 and the second collapse parameter calculation unit 212.

[0067] Moreover, data indicating the height h of the object is input to the first collapse parameter calculation unit 211 and the second collapse parameter calculation unit 212 (step S132).

[0068] The height h of the object is set in advance. For example, when the object is an automobile, a value of a general automobile height or a value with a margin therefor is input to the change map generating unit 21 as the height h of the object.

[0069] The first and second collapse parameter calculation units 211 and 212 calculate the collapse parameters (step S133). In step S133, the first collapse parameter calculation unit 211 calculates the collapse parameters based on the incident angle θ AUsing the height h of the object and the above equation (1), the amount of inclination l of the object in image A is calculated. A In addition, the first tilt parameter calculation unit 211 calculates the range azimuth angle α A is set as the falling direction of the object. The first falling parameter calculation unit 211 sets the obtained falling amount and falling direction as the first falling parameter. Note that, when a plurality of objects exist in the image A, the first falling parameter calculation unit 211 determines the falling amount and falling direction of each object, and includes each of the falling amount and falling direction in the first falling parameter.

[0070] In step S133, the second collapse parameter calculation unit 212 calculates the incident angle θ B Using the height h of the object and the above formula (1), the amount of inclination l of the object in image B is calculated. B In addition, the second tilt parameter calculation unit 212 calculates the range azimuth angle α B is set as the falling direction of the object. The second falling parameter calculation unit 212 sets the obtained falling amount and falling direction as the second falling parameter. When multiple objects exist in the image B, the second falling parameter calculation unit 212 determines the falling amount and falling direction of each object, and includes each of the falling amount and falling direction in the second falling parameter.

[0071] In addition, when an optical image is used as the observation image, the first tilt parameter calculation unit 211 calculates the range azimuth angle α A The second tilt parameter calculation unit 212 determines the direction of 180 degrees with respect to the range azimuth angle α B The direction of 180 degrees with respect to the second collapse parameter is determined as the collapse direction in the second collapse parameter.

[0072] The first expansion unit 311 and the second expansion unit 312 expand the object in the object map (image A or image B) (step S134). In step S134, the first expansion unit 311 expands the object in image A by a collapse amount l in the collapse direction included in the second collapse parameter. B The second expansion unit 312 expands the object in the image B by a collapse amount l in the collapse direction included in the first collapse parameter. A Just expand it.

[0073] The map generating unit 411 superimposes image A in which the object is expanded (first object map 112: see FIG. 6) and image B in which the object is expanded (second object map 122: see FIG. 6) (step S135).

[0074] The map generating unit 411 determines whether the object has changed based on the overlapping of the object in the superimposed image created in the process of step S135. For example, the map generating unit 411 determines whether the object has changed by comparing the first object map 112 and the second object map 122 for each pixel (pixel by pixel). Then, as illustrated in FIG. 7, the change map generating unit 411 determines that an object that existed in image A but does not exist in image B is a disappeared object (changed object). The map generating unit 411 also determines that an object that did not exist in image A but exists in image B is a newly appeared object (changed object). The map generating unit 411 determines that other objects are unchanged objects.

[0075] The map generating unit 411 generates a change map 140 (see FIG. 8) by reflecting the result of the determination as to whether or not a change has occurred in the superimposed image created in the processing of step S135 (step S136).

[0076] Data indicating the width of the object is input to noise elimination unit 511 (step S137). The width of the object is set in advance. For example, if the object is an automobile, a value of a typical automobile width or a value with some margin thereon is input to noise elimination unit 511 as the width of the object. Note that the process of step S137 does not have to be executed at the timing shown in FIG. 11. In other words, it is sufficient that the width of the object is input before the process of step S138 is started.

[0077] The noise elimination unit 511 performs an opening process on the change map 140, and outputs the change map (final change map) 150 from which noise has been removed (step S138). In the process of step S138, the noise elimination unit 511 performs a contraction process in the opening process to contract the object by a number of pixels according to the size (specifically, the width) of the object. The number of pixels to be contracted is determined in advance according to the size of the object. In other words, the number is set to a number that allows a group of pixels that should not be determined to be the object to be removed. As an example, when the maximum width of the object is three pixels, the noise elimination unit 511 performs a contraction process twice in the opening process so that blocks of less than three pixels, i.e., blocks of two pixels or less in size, are removed.

[0078] The coincidence calculation unit 22 calculates the degree of coincidence between image A and image B based on change map 150 generated by change map generation unit 21 (step S14). In the process of step S14, the coincidence calculation unit 22 calculates, for example, the ratio of the number of pixels of a non-changed region (an elliptical region surrounded by a solid line in the example shown in FIG. 3) to the total number of pixels of change map 150 as the degree of coincidence. In other words, the coincidence calculation unit 22 calculates the ratio of the size of the non-changed region where the object has not changed to the total area of ​​image B as the degree of coincidence.

[0079] The matching degree determination unit 23 determines whether or not the image pair (image A and image B) is an object-matching image pair based on the degree of matching calculated by the matching degree calculation unit 22 (step S15). For example, when the degree of matching exceeds a predetermined value, the matching degree determination unit 23 determines the image pair as an object-matching image pair. Then, the matching degree determination unit 23 outputs the object-matching image pair.

[0080] As described above, the image processing device of this embodiment determines the degree of match (similarity) of an image pair by taking into consideration the state of an object in an observed image (appearance or disappearance of an object). Then, the image processing device adopts an image pair with a high degree of match as an object-matched image pair. As a result, the image processing device can collect a large number of images of the same area, which increases the reliability of analysis of ground surface displacement, etc.

[0081] Embodiment 2. Fig. 12 is a block diagram showing an example of the configuration of an image pair determination unit in an image processing device of the second embodiment. The configuration of the image processing device of the second embodiment is as shown in Fig. 1. Image pair determination unit 20B shown in Fig. 12 includes a change map generation unit 21, a changed area identification unit 24, and a mask processing unit 25. Note that image pair determination unit 20B shown in Fig. 12 corresponds to image pair determination unit 20 shown in Fig. 1.

[0082] The change map generating unit 21 performs the same processing as in the first embodiment. The changed area identifying unit 24 identifies a changed area (an area where an object has disappeared or appeared) in the pair of object maps, i.e., the image pair (image A and image B), using the change map generated by the change map generating unit 21. The mask processing unit 25 performs mask processing on the changed areas in image A and image B.

[0083] FIG. 13 is an explanatory diagram for explaining mask processing. The upper left side of FIG. 13 shows an example of image A (first object map 111) obtained from a first observation image. Three automobiles 91, 92, and 93 exist in image A. The lower left side of FIG. 13 shows an example of image B (second object map 121) obtained from a second observation image. Two automobiles 93 and 94 exist in image B. The time when the first observation image was obtained is time t1. The time when the second observation image was obtained is time t2. At time t2, automobiles 91 and 92 that existed at time t1 have disappeared. Furthermore, a new automobile 94 appears at time t2.

[0084] In this example, the first object map 111 and the second object map 121 correspond to an image of a parking lot 120.

[0085] In image A, the changed area is an area including the area where automobiles 91 and 92 exist. In image B, the changed area is an area including the area where automobile 94 exists. Then, similar to the first embodiment, a change map is generated using image A and image B. Referring to change map 150 in FIG. 3, an ellipse surrounded by a solid line in change map 150 indicates an area where automobile 93, which did not change from time t1 to time t2, exists. In other words, it indicates an area where there is no change. A black ellipse indicates an area where a newly appeared automobile 94 exists. An ellipse surrounded by a dashed line indicates an area where vanished automobiles 91 and 92 exist. In other words, a black ellipse and an ellipse surrounded by a dashed line indicate changed areas.

[0086] In this embodiment, mask processing is performed on the area in image A where automobiles 91 and 92 exist. Also, mask processing is performed on the area in image B where automobile 94 exists. As shown in Fig. 13, the mask processing on image A forms a mask area M1 in the masked image A (image 113). The mask processing on image B forms a mask area M2 in the masked image B (image 123).

[0087] Next, the operation of the image processing apparatus according to the second embodiment will be described with reference to the flowchart of FIG.

[0088] The image processing device 1 inputs pairs of observed images (step S11). In the image processing device 1, the object map creation unit 10 creates pairs of object maps from each of the observed images, similar to the first embodiment (step S12). The change map generation unit 21 in the image pair determination unit 20B generates a change map from the pair of object maps, similar to the first embodiment (step S13).

[0089] The changed area identifying section 24 in the image pair determining section 20B identifies a changed area (an area where an object has disappeared or appeared) in the image pair (image A and image B) by referring to the change map generated by the change map generating section 21 (step S21). The mask processing section 25 performs mask processing on the changed areas in the images A and B (step S22).

[0090] In the process of step S22, mask processing unit 25 replaces, for example, the values ​​of all pixels in the changed regions in images A and B with predetermined default values. The regions whose values ​​have been replaced are mask regions M1 and M2 in FIG.

[0091] Mask processing unit 25 may replace pixels of the changed regions in images A and B with pixels of a background image in a previously obtained object map. For image A, mask processing unit 25 may use a background image corresponding to the trajectory from which the observation image on which image A is based was obtained, and for image B, mask processing unit 25 may use a background image corresponding to the trajectory from which the observation image on which image B is based was obtained. That is, mask processing unit 25 may use different background images depending on the trajectory. In that case, when multiple background images are obtained for each trajectory, mask processing unit 25 may use the minimum value in the multiple background images for each pixel.

[0092] The observed image on which image A was based is an observed image that image A appears in. The observed image on which image B was based is an observed image that image B appears in.

[0093] The mask processing unit 25 in the image pair determination unit 20B outputs the image A and image B that have been subjected to the mask processing as an object-matching image pair (step S23).

[0094] The image processing device of this embodiment performs mask processing so as to increase the degree of match between image pairs, taking into account the state of the object in the observed image (appearance or disappearance of the object). The masked area is neither a changed area nor a non-changed area. Therefore, the mask processing makes the changed area of ​​one image relatively small compared to the non-changed area where the object has not changed. Then, the image processing device adopts the image pair that has been subjected to the mask processing as an object-matched image pair. As a result, the image processing device can collect a large number of images capturing the same area.

[0095] Embodiment 3. Fig. 15 is a block diagram showing an example of the configuration of an image pair determination unit in an image processing device of the third embodiment. The configuration of the image processing device of the third embodiment is as shown in Fig. 1. The image pair determination unit 20C shown in Fig. 15 includes a change map generation unit 21, a same trajectory data collection unit 26, a changed area identification unit 27, a changed area replacement unit 28, and an object map storage memory 30. Note that the image pair determination unit 20C shown in Fig. 15 corresponds to the image pair determination unit 20 shown in Fig. 1.

[0096] The change map generating unit 21 performs the same process as in the first embodiment. However, in this embodiment, as described later, the change map generating unit 21 generates a large number of change maps. The large number of change maps are stored in the object map storage memory 30. Specifically, the object map storage memory 30 temporarily stores pairs of object maps and change maps generated from the pairs in association with each other.

[0097] For example, a plurality of pairs each consisting of image A and a plurality of other images are stored in the object map storage memory 30. Also, the object map storage memory 30 stores a change map created from image A and each of the plurality of other images.

[0098] The same trajectory data collection unit 26 extracts each of a plurality of object maps (assumed to be image B') obtained from observation images taken on the same trajectory (assumed to be trajectory B) and each change map corresponding to image B' from the object map storage memory 30. Strictly speaking, the change map corresponding to image B' is a change map generated from a pair including image B' (in this embodiment, image A and image B').

[0099] The changed region identifying section 27 identifies non-changed regions in the image B' extracted from the object map storage memory 30 by using the change map corresponding to the image B'.

[0100] The changed region replacing unit 28 synthesizes the non-changed regions in the multiple images B' to generate image B (second object map 121).

[0101] FIG. 16 is an explanatory diagram for explaining a method of generating one image in an image pair. The upper left side of FIG. 16 shows an example of image A (first object map 115) created by object map creation unit 10. Three automobiles 91, 92, and 93 exist in image A. The lower left side of FIG. 13 shows an example of image B' (object maps 124A, 124B) for generating desired image B (second object map 125). Two images B' are illustrated in FIG. 16.

[0102] In the example shown in FIG. 16, two automobiles 93, 94 are present in image B' (object map 124A). Compared with image A, automobile 94 in image B' (object map 124A) is a newly appeared automobile. Also, automobile 92 has disappeared in image B' (object map 124A). That is, the region in image B' (object map 124A) where automobile 94 exists and the region where automobile 92 has disappeared are changed regions. Automobile 93 is also present in image A. That is, the region including automobile 93 in image B' (object map 124A) is unchangeable region R1.

[0103] Image B' (object map 124B) has three automobiles 91, 92, and 95. Compared to image A, automobile 95 in image B' (object map 124B) is a newly appeared automobile. That is, in image B' (object map 124B), the area where automobile 95 exists is a changed area. Automobiles 91 and 92 also exist in image A. That is, in image B' (object map 124B), the area including automobiles 91 and 92 is a non-changed area R2.

[0104] The non-changed region R1 in image B' (object map 124A) and the non-changed region R2 in image B' (object map 124B) are combined. Specifically, one image (e.g., object map 124A) is selected from the multiple images B' input from the object map storage memory 30, and the changed region in the selected image is replaced with the non-changed region (e.g., non-changed region R2) in the other image (e.g., object map 124B). The non-changed region R1 in one image (e.g., object map 124A) is used as is. The changed and non-changed regions in each image B' are specified by the change map corresponding to that image B'.

[0105] Next, the operation of the image processing apparatus of the third embodiment will be described with reference to the flowchart of FIG.

[0106] Image processing device 1 inputs pairs of observed images (step S11). In image processing device 1, object map creation unit 10 creates pairs of object maps from each of the observed images, similar to the first embodiment (step S12). Change map generation unit 21 in image pair determination unit 20C generates a change map from the pair of object maps, similar to the first embodiment (step S13). However, in this embodiment, change map generation unit 21 generates multiple pairs, for example, consisting of image A and each of multiple other images. Change map generation unit 21 stores the change map created from image A and each of the multiple other images, and each of the multiple other images, in object map storage memory 30.

[0107] The same trajectory data collection unit 26 extracts (collects) from the object map storage memory 30 a plurality of images B', each of which is paired with image A, and change maps corresponding to images A and B' (change maps generated from images A and B') (step S31). Referring to the example shown in Fig. 16, the same trajectory data collection unit 26 extracts object map 124A and object map 124B, as well as a change map generated from image A and object map 124A and a change map generated from image A and object map 124B.

[0108] In addition, the multiple images B' paired with image A are images of the same area as the shooting area of ​​image A, and the original observation images were obtained from the same trajectory (different from the trajectory for image A).

[0109] Changed area identifying section 27 selects one of the multiple images B' extracted in the processing of step S31 (step S32). In the example shown in FIG. 16, object map 124A is selected. Hereinafter, selected image B' is referred to as object map 124A. Furthermore, changed area identifying section 27 uses a change map corresponding to selected object map 124A to identify a changed area in object map 124A (in the example shown in FIG. 16, an area other than non-changed area R1) (step S33).

[0110] The changed area replacement unit 28 checks the non-changed areas in each of the multiple images B' other than the selected object map 124A. Specifically, the changed area replacement unit 28 identifies the image B' including a non-changed area that can effectively replace the changed area in the object map 124A. An image including a non-changed area that can effectively replace a changed area is, for example, an image having a non-changed area that can replace the widest area in the changed area with a non-changed area. In other words, based on the number of pixels in the changed area in the object map 124A, the image is the image having the non-changed area with the largest number of pixels.

[0111] The changed region replacing unit 28 replaces the changed region in the object map 124A with the unchangeable region in the specified image B' (step S34). The image B' that has been subjected to such replacement processing is the desired image B.

[0112] The changed area replacing section 28 in the image pair determining section 20C outputs image A and image B as an object-matching image pair (step S35).

[0113] The image processing device of this embodiment converts a changed area of ​​one image into a non-changed area by utilizing a non-changed area in another image obtained by observing the same observation area as the observation area of ​​the first image. That is, the number of pixels in the non-changed area is increased in the first image. In other words, the changed area of ​​the first image becomes smaller relative to the non-changed area in which the object has not changed. Then, the image processing device sets the image with the increased number of pixels in the non-changed area as one image in the object-matched image pair. As a result, the image processing device can collect multiple images of the same area, which increases the reliability of analysis of ground surface displacement, etc.

[0114] Fig. 18 is a block diagram showing an example of the configuration of an information processing device capable of implementing the functions of the image processing device of each of the above-mentioned embodiments. The information processing device shown in Fig. 18 includes one or more processors such as CPUs (Central Processing Units), a program memory 1002, and a memory 1003. Fig. 18 shows an information processing device having one processor 1001.

[0115] The program memory 1002 is, for example, a non-transitory computer readable medium. The non-transitory computer readable medium includes various types of tangible storage media. For example, a semiconductor storage medium such as a flash ROM (Read Only Memory) or a magnetic storage medium such as a hard disk can be used as the program memory 1002. The program memory 1002 stores image processing programs for implementing the functions of each block (object map creation unit 10, image pair determination units 20, 20A, 20B, 20C) in the image processing device of the above embodiment.

[0116] The processor 1001 realizes the functions of the image processing device by executing processing according to an image processing program stored in the program memory 1002. When multiple processors are installed, the multiple processors can work together to realize the functions of the image processing device.

[0117] For example, a RAM (Random Access Memory) can be used as the memory 1003. The memory 1003 stores temporary data and the like that is generated when the image processing device is executing a process. It is also possible to assume a configuration in which an image processing program is transferred to the memory 1003, and the processor 1001 executes processing based on the image processing program in the memory 1003. Note that the program memory 1002 and the memory 1003 may be integrated.

[0118] Fig. 19 is a block diagram showing the main parts of an image processing device. The image processing device 60 shown in Fig. 19 includes a changed area detection means 61 (changed area detection unit: in the second and third embodiments, this is realized by the change map generation unit 21) that detects a changed area in a second image (e.g., image B) obtained by observing an observation area that is the same as the observation area of ​​an object in a first image (e.g., image A) where the object has changed, an image correction means 62 (image correction unit: in the second embodiment, this is realized by the mask processing unit 25, and in the third embodiment, this is realized by the changed area replacement unit 28) that performs a correction process to make the changed area in the second image relatively smaller than the non-changed area where the object has not changed, and an image pair output means 63 (image pair output unit: in the second embodiment, this is realized by the mask processing unit 25, and in the third embodiment, this is realized by the image pair determination unit 20C and the changed area replacement unit 28 in the image pair determination unit 20C) that outputs an image pair of the second image and the first image processed by the image correction means 62.

[0119] Fig. 20 is a block diagram showing the main parts of an image processing device of another aspect. The image processing device 70 shown in Fig. 20 includes a changed area detection means 71 (changed area detection unit: in the first embodiment, this is realized by the change map generation unit 21) for detecting a changed area in which an object in a first image (e.g., image A) has changed in a second image (e.g., image B) obtained by observing the same observation area as the observation area of ​​the first image, a coincidence calculation means 72 (coincidence calculation unit: in the first embodiment, this is realized by the coincidence calculation unit 22) for calculating the coincidence between the first image and the second image based on the size of a non-changed area in which the object in the second image has not changed, and an image pair output means 73 (image pair output unit: in the first embodiment, this is realized by the coincidence determination unit 23) for outputting an image pair of the second image and the first image when the coincidence exceeds a predetermined value.

[0120] Although a part or all of the above-described embodiments can be described as follows, the present invention is not limited to the following configurations.

[0121] (Note 1) A change area detection means for detecting a change area in a second image obtained by observing an observation area that is the same as the observation area of ​​an object in a first image, where the object has changed; an image correction means for performing a correction process to make a changed area in the second image smaller relative to a non-changed area in which the object is not changed; an image pair output means for outputting an image pair of the second image and the first image processed by the image correction means; An image processing device comprising:

[0122] (Note 2) The image correction means replaces pixel values ​​of the changed area in the second image with default values. The image processing device of Appendix 1.

[0123] (Note 3) The image correction means replaces the changed area in the second image with a non-changed area in another image obtained by observing the same observation area as the observation area in the first image. The image processing device of Appendix 1.

[0124] (Note 4) A change area detection means for detecting a change area in a second image obtained by observing an observation area that is the same as the observation area of ​​the first image, in which the object in the first image has changed; a matching degree calculation means for calculating a matching degree between the first image and the second image based on a size of a non-changing area in the second image where the object is not changed; an image pair output means for outputting an image pair of the second image and the first image when the degree of match exceeds a predetermined value; An image processing device comprising:

[0125] (Note 5) The coincidence calculation means calculates the ratio of the number of pixels in a non-changing area to the number of pixels in an entire area in the second image as the coincidence. The image processing device of Appendix 4.

[0126] (Additional Note 6) The change area detection means an image deformation means (realized in the embodiment by a first expansion means 311 and a second expansion means 312) for deforming object presence areas in two object presence images (e.g., a first object map 111 and a second object map 121) in which one or more objects are present and which are images obtained from each of the two observation images based on the observation angles (e.g., range azimuth and incidence angle) of each of the two observation images and the size of the object appearing in each of the two observation images (e.g., the height of the object) to generate two deformed images (e.g., a first object map 112 and a second object map 122); an image generating means (realized by a map generating unit 411 in the embodiment) for generating a composite image by synthesizing the two deformed images, determining a change in the object between the two object-present images using the composite image, and generating an image (e.g., a change map 140) capable of identifying the determined change; 6. An image processing device according to any one of claims 1 to 5,

[0127] (Appendix 7) Detecting a change area in a second image obtained by observing an observation area that is the same as the observation area of ​​the first image, where the object in the first image has changed; performing a correction process to make a changed area in the second image smaller relative to a non-changed area in which the object has not changed; outputting an image pair of the corrected second image and the first image; Image processing methods.

[0128] (Appendix 8) In the correction process, pixel values ​​of the changed area in the second image are replaced with default values. 8. The image processing method according to claim 7.

[0129] (Appendix 9) In the correction process, the changed area in the second image is replaced with a non-changed area in another image obtained by observing the same observation area as the observation area in the first image. Image processing method of Appendix 7.

[0130] (Appendix 10) For an object in a first image, a change area in which the object has changed in a second image obtained by observing the same observation area as the observation area in the first image is detected; Calculating a degree of matching between the first image and the second image based on a size of a non-changed region in which the object in the second image has not changed; When the degree of match exceeds a predetermined value, an image pair of the second image and the first image is output. Image processing methods.

[0131] (Appendix 11) The ratio of the number of pixels in the non-changed area to the number of pixels in the entire area in the second image is calculated as the degree of match. Image processing method of Appendix 10.

[0132] (Appendix 12) When detecting change areas, based on the observation angle of each of the two observation images and the size of the object appearing in each of the two observation images, deforming object presence regions in two object presence images, which are images obtained from each of the two observation images and in which one or more objects are present, to generate two deformed images; The two deformed images are combined to generate a combined image, and the combined image is used to determine a change in the object between the two object-present images, and an image is generated in which the determined change can be identified. 12. The image processing method according to any one of claims 7 to 11.

[0133] (Appendix 13) A computer-readable recording medium storing an image processing program, The image processing program is installed in a computer. A process of detecting a changed region in a second image obtained by observing an observation area that is the same as the observation area of ​​an object in a first image, where the object has changed; a correction process for making a changed area in the second image smaller relative to a non-changed area in which the object is not changed; and outputting an image pair of the corrected second image and the first image. Recording medium.

[0134] (Appendix 14) A computer-readable recording medium storing an image processing program, The image processing program is installed in a computer. A process of detecting a changed region in a second image obtained by observing an observation area that is the same as the observation area of ​​an object in a first image, where the object has changed; A process of calculating a degree of matching between the first image and the second image based on a size of a non-changed area in which the object in the second image has not changed; and executing a process of outputting an image pair of the second image and the first image when the degree of match exceeds a predetermined value. Recording medium.

[0135] (Appendix 15) To the computer, A process of detecting a changed region in a second image obtained by observing an observation area that is the same as the observation area of ​​an object in a first image, where the object has changed; a correction process for making a changed area in the second image smaller relative to a non-changed area in which the object is not changed; an image processing program for executing a process of outputting an image pair of the corrected second image and the first image.

[0136] (Appendix 16) To the computer, A process of detecting a changed region in a second image obtained by observing an observation area that is the same as the observation area of ​​an object in a first image, where the object has changed; A process of calculating a degree of matching between the first image and the second image based on a size of a non-changed area in which the object in the second image has not changed; and outputting an image pair of the second image and the first image when the degree of match exceeds a predetermined value.

[0137] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Explanation of symbols]

[0138] 1 Image processing device 10. Object Map Creation Section 20 Image pair judgment unit 20A, 20B, 20C Image pair determination unit 21 Change map generation unit 22 Matching degree calculation section 23 Matching degree determination section 24 Change area identification section 25 Mask processing section 26 Same-orbit data collection unit 27 Change area identification section 28 Change Area Replacement Section 30 Object map storage memory 60,70 Image processing device 61,71 Change area detection means 62 Image correction means 63,73 Image pair output means 72 Matching degree calculation means 100 satellites 211 First collapse parameter calculation unit 212 Second Collapse Parameter Calculation Unit 311 First expansion section 312 Second expansion section 411 Map Generation Unit 511 Noise Cancellation Unit 1001 Processor 1002 Program Memory 1003 Memory

Claims

1. a change region detection means for detecting a change region in a second image obtained by observing an observation region that is the same as the observation region of the first image, in relation to an object in the first image; an image correction means for performing a correction process to make a changed area in the second image smaller relative to a non-changed area in which the object is not changed; an image pair output unit that outputs an image pair of the second image and the first image processed by the image correction unit, The change region detection means an image transformation means for transforming object presence areas in two object presence images, each of which is an image obtained from each of the two observation images and contains one or more objects, based on an observation angle of each of the two observation images and a size of an object appearing in each of the two observation images, to generate two transformed images; and an image generating means for generating a composite image by combining the two deformed images, determining a change in the object between the two object-present images using the composite image, and generating an image in which the determined change can be identified. Image processing device.

2. The image correction means replaces pixel values ​​of the changed area in the second image with default values.

2. The image processing device according to claim 1.

3. The image correction means replaces the changed area in the second image with a non-changed area in another image obtained by observing the same observation area as the observation area in the first image.

2. The image processing device according to claim 1.

4. A computer comprising: Detecting a change region in a second image in which the object in the first image has changed, the second image being obtained by observing the same observation region as the observation region in the first image; performing a correction process to make a changed area in the second image smaller relative to a non-changed area in which the object has not changed; outputting an image pair of the corrected second image and the first image; When detecting a change region in the second image, generating two deformed images by deforming object presence regions in two object presence images, the object presence images being images obtained from the two observation images and each of which contains one or more objects, based on an observation angle of each of the two observation images and a size of an object appearing in each of the two observation images; The two deformed images are combined to generate a combined image, and the combined image is used to determine a change in the object between the two object-present images, and an image is generated in which the determined change can be identified. Image processing methods.

5. In the correction process, the computer replaces pixel values ​​of the changed area in the second image with default values.

5. The image processing method according to claim 4.

6. In the correction process, the computer replaces the changed area in the second image with a non-changed area in another image obtained by observing the same observation area as the observation area in the first image.

5. The image processing method according to claim 4.

Images