Displacement extraction device, displacement extraction system, displacement extraction method, and computer program

Abstract

To provide a displacement extraction device capable of increasing the possibility of providing a user with a highly accurate displacement extraction result.SOLUTION: A displacement extraction device 3 includes a target image acquisition unit 33 that acquires a target image which is an image of a target area subject to displacement extraction, a reference image acquisition unit 35 that acquires a reference image which is an image of the target area and has capturing date and time different from those of the target image, an image conversion unit 38 that generates a converted image by converting a first image, which is one of the reference image and the target image, into an image of the image type of a second image, which is the other image, when the image type of the reference image and the image type of the target image are different, a displacement extraction unit 40 that extracts a displacement by comparing the second image with the converted image, and an output unit 41 that outputs the extraction result by the displacement extraction unit 40. The image type includes a SAR image or an optical sensor.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a displacement extraction device, a displacement extraction system, a displacement extraction method, and a computer program that extract displacement using an image. [Background technology]

[0002] In recent years, the use of sensor images, which are images acquired by sensors, has been considered for grasping the situation of disasters such as earthquakes, heavy rains, and fires, as well as for disaster prevention measures and monitoring specific locations. Examples of sensors include passive sensors that passively observe electromagnetic waves and active sensors that transmit electromagnetic waves and observe the reflected electromagnetic waves. An example of a passive sensor is an optical sensor known as an image sensor, and an example of an active sensor is a synthetic aperture radar (SAR).

[0003] Patent Document 1 discloses a technology that uses SAR images, which are images obtained by observations using SAR mounted on artificial satellites, to grasp the extent of damage. In the technology described in Patent Document 1, a data analysis center extracts areas of change on the earth's surface from the difference between SAR images taken under normal circumstances and SAR images taken after a disaster has occurred. The data analysis center then overlays the extracted areas of change on map information to create a change extraction map and provides it to an administrative agency. This allows the administrative agency to grasp the extent of the damage. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2008 / 016153 Summary of the Invention [Problem to be solved by the invention]

[0005] The system described in Patent Document 1 calculates displacement from the difference between SAR images taken during normal times and those taken after a disaster. Therefore, both SAR images taken during normal times and those taken after a disaster are required. However, satellite observations are generally performed from polar orbits such as quasi-recurrent orbits, and image data of the same location is not always available. Therefore, SAR images taken during both normal times and after a disaster may not be available. Patent Document 1 describes simulating images taken by SAR using three-dimensional geographic information data from a three-dimensional geographic information system or dielectric constant parameters for each feature instead of SAR images taken during normal times. Even for the same structure, the dielectric constant varies depending on the material, and the task of aligning dielectric constant parameters for each feature material is enormous. Therefore, such data is only available for certain areas, such as urban areas.

[0006] For this reason, the technology described in Patent Document 1 cannot provide the user with displacement extraction results if it is unable to obtain three-dimensional geographic information data from a three-dimensional geographic information system or data such as permittivity parameters for each feature for an area for which a user requests displacement extraction. Even if three-dimensional geographic information data or permittivity parameters for each feature are obtained from a three-dimensional geographic information system, simulated images generated from these data are not actually observed data and therefore have large errors, resulting in large displacement errors when displacements are calculated using simulated images. For this reason, the technology described in Patent Document 1 has a problem in that it can only provide users with highly accurate displacement extraction results under limited conditions, making it unlikely that highly accurate displacement extraction results can be provided.

[0007] The present disclosure has been made in view of the above, and aims to provide a displacement extraction device that can increase the possibility of providing a user with a highly accurate displacement extraction result. [Means for solving the problem]

[0008] In order to solve the above-described problems and achieve the object, the displacement extraction device according to the present disclosure includes a target image acquisition unit that acquires a target image, which is an image of a target area that is a target for displacement extraction, and the image type of the image of the target area is a sensor image of at least one of an SAR image and an optical image acquired by a synthetic aperture radar, and a reference image acquisition unit that acquires a reference image, which is an image type of the image of the target area is a sensor image of at least one of an SAR image and an optical image acquired by a synthetic aperture radar, and which is captured at a different date and time from the target image. , painting Image conversion unit and ,strange The apparatus includes a displacement extraction unit that extracts the displacement of a target object, and an output unit that outputs the extraction result by the displacement extraction unit. When the target image acquisition unit can acquire target images that are optical images and target images that are SAR images, the target image acquisition unit selects an image type corresponding to the type of target object to be detected for displacement from among the target images that can be acquired. Priority type is When the reference image acquisition unit can acquire a reference image that is an optical image and a reference image that is an SAR image, the reference image acquisition unit acquires the following among the reference images that can be acquired: Priority Type A reference image is acquired. When the image type of the target image acquired by the target image acquisition unit differs from the image type of the reference image acquired by the reference image acquisition unit, the image conversion unit generates a converted image by performing image conversion to convert an image of the target image acquired by the target image acquisition unit and the reference image acquired by the reference image acquisition unit, which has an image type different from the priority type, into an image of the priority type; when the image type of the target image acquired by the target image acquisition unit differs from the image type of the reference image acquired by the reference image acquisition unit, the displacement extraction unit extracts the displacement by comparing the converted image with an image of the priority type from the target image acquired by the target image acquisition unit and the reference image acquired by the reference image acquisition unit; when the image type of the target image acquired by the target image acquisition unit is the same as the image type of the reference image acquired by the reference image acquisition unit, the displacement extraction unit extracts the displacement by comparing the target image acquired by the target image acquisition unit with the reference image acquired by the reference image acquisition unit. The displacement extraction device according to the present disclosure includes a target image acquisition unit that acquires a target image, which is an image of a target area from which displacement is extracted, and a reference image acquisition unit that acquires a reference image, which is an image of the target area and has an image capture date and time different from that of the target image. an image type determination unit that determines a priority type, which is an image type for which image conversion is not performed when the image types of the target image and the reference image are different, according to the type of object for which displacement is to be detected; The displacement extraction device further includes: when the image type of the reference image and the image type of the target image are different, Images of a different image type from the priority type are treated as images of the priority type. an image conversion unit that generates a converted image by converting If the image type of the target image is different from the image type of the reference image, the displacement is extracted by comparing the converted image with the image of the priority type among the reference image and the target image, and if the image type of the target image is the same as the image type of the reference image, the displacement is extracted by comparing the target image acquired by the target image acquisition unit with the reference image acquired by the reference image acquisition unit. and an output unit that outputs the extraction result by the displacement extraction unit. priority The type is SAR image acquired by synthetic aperture radar or optical image acquired by optical sensor, The target image is a SAR image or an optical image, and the reference image is a SAR image or an optical image. do. The displacement extraction device according to the present disclosure also includes a target image acquisition unit that acquires a target image, which is an image of a target region from which displacement is extracted; a reference image acquisition unit that acquires a reference image, which is an image of the target region and has an image capture date and time different from that of the target image; and an image conversion unit that generates a converted image by converting a first image, which is one of the reference image and the target image, into an image of the image type of a second image, which is the other image, when the image type of the reference image and the target image are different. ,strange and an output unit that outputs the extraction result by the displacement extraction unit, wherein the image type is a SAR image acquired by a synthetic aperture radar or an optical image acquired by an optical sensor, the image conversion unit acquires a probability map that indicates a distribution of the probability of the image conversion, and the displacement extraction unit If the image type of the target image is different from the image type of the reference image, The displacement is extracted using the comparison result between the second image and the transformed image and the likelihood map. If the image type of the target image and the image type of the reference image are the same, the target image acquired by the target image acquisition unit is compared with the reference image acquired by the reference image acquisition unit to extract the displacement. do. The displacement extraction device according to the present disclosure also includes a target image acquisition unit that acquires a target image, which is an image of a target area from which displacement is extracted; a reference image acquisition unit that acquires a reference image, which is an image of the target area and has a different image capture date and time from the target image; and an image conversion unit that generates a converted image by converting a first image, which is one of the reference image and the target image, into an image of the image type of a second image, which is the other image, when the image type of the reference image and the target image are different. If the image type of the target image is different from the image type of the reference image, Displacement is extracted by comparing the second image with the transformed image If the image type of the target image and the image type of the reference image are the same, the target image acquired by the target image acquisition unit is compared with the reference image acquired by the reference image acquisition unit to extract the displacement. and an output unit that outputs the extraction result by the displacement extraction unit, the image type being an SAR image acquired by a synthetic aperture radar or an optical image acquired by an optical sensor, the image conversion unit acquiring a probability map that indicates the distribution of the probability of the image conversion, and the output unit outputting the extraction result in association with the probability map. [Effects of the Invention]

[0009] The displacement extraction device according to the present disclosure has an effect of increasing the possibility of providing a highly accurate displacement extraction result to a user. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating a configuration example of a displacement extraction system according to a first embodiment. [Figure 2] 1 is a flowchart showing an example of a displacement extraction process in the displacement extraction device according to the first embodiment. [Figure 3] FIG. 10 is a diagram showing an example of image type correspondence information according to the first embodiment; [Figure 4] FIG. 1 shows an example of the configuration of a learning device according to a first embodiment. [Figure 5] A simplified diagram showing an example of a neural network. [Figure 6] FIG. 10 is a diagram showing a configuration example of an image conversion unit according to the first embodiment when image conversion is performed by machine learning. [Figure 7] FIG. 1 is a diagram showing an example of the configuration of a computer system that realizes a displacement extraction device according to a first embodiment. [Figure 8] FIG. 10 is a diagram illustrating a configuration example of a displacement extraction system according to a second embodiment. [Figure 9] FIG. 10 is a diagram showing an example of a target date and time designation method when the target date and time information in the second embodiment is information indicating the date and time of shooting. [Figure 10] Flowchart showing an example of image acquisition processing according to the second embodiment [Figure 11] FIG. 10 is a diagram showing a configuration example of a displacement extraction system according to a first modification of the second embodiment; [Figure 12] FIG. 10 is a diagram illustrating a configuration example of a displacement extraction system according to a third embodiment. [Figure 13] FIG. 10 is a diagram illustrating a configuration example of a displacement extraction system according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] A displacement extraction device, a displacement extraction system, a displacement extraction method, and a computer program according to embodiments will be described in detail below with reference to the accompanying drawings.

[0012] Embodiment 1 FIG. 1 is a diagram illustrating a configuration example of a displacement extraction system according to a first embodiment. The displacement extraction system 1 of this embodiment includes a displacement extraction device 3 and a user terminal 2. The displacement extraction device 3 extracts displacement using a sensor image and outputs the displacement extraction result. In the example illustrated in FIG. 1, the displacement extraction device 3 acquires, from the user terminal 2, a target image, which is a sensor image capturing a target area from which displacement is to be extracted, and a reference image, which is a sensor image serving as a reference for displacement extraction, extracts displacement using the target image and the reference image, and outputs the displacement extraction result to the user terminal 2. The sensor image may be an SAR image, an optical image, or a combination of an SAR image and an optical image.

[0013] The user terminal 2 is a terminal that can be operated by a user who requests displacement extraction. The user may be a local government, a country, an organization or corporation that manages roads, facilities, rivers, forests, etc., or an individual. If the user is not an individual, the user terminal 2 is operated by an operator or the like of the user. Hereinafter, the term "input by the user to the user terminal 2" will also be used to refer to the case where input is made by a person such as an operator of the user when the user is not an individual. The user terminal 2 includes an image acquisition unit 21, an image storage unit 22, an input acceptance unit 23, a transmission / reception unit 24, and a display unit 25.

[0014] The image acquisition unit 21 acquires sensor images from the image providing system 4 and stores the sensor images in the image storage unit 22. For example, if the sensor images are images acquired by satellite observation, the image providing system 4 is a system that provides satellite observation images. If the sensor images are images acquired by an aircraft, the image providing system 4 is a system that provides images acquired by the aircraft. The aircraft may be a manned aircraft or an unmanned aircraft called a drone. Furthermore, if the sensor images are images acquired by an aircraft and the observation by the aircraft was performed at the request of a user, the image acquisition unit 21 may acquire the sensor images from a sensor mounted on the aircraft. Furthermore, the image providing system 4 may be an external system connected to a wide area communication network such as the Internet, or may be a system managed by the user and connected to the user terminal 2 via a local area network. In the latter case, the image providing system 4 may acquire and store sensor images from the external system in advance.

[0015] The input receiving unit 23 receives input from the user and outputs input information indicating the received input to the transmitting / receiving unit 24. For example, the input receiving unit 23 receives input of designation information for specifying conditions for displacement extraction, and outputs the input designation information to the transmitting / receiving unit 24. The designation information includes area information that designates an area of ​​the target image that is to be subjected to displacement extraction. Furthermore, as will be described later, when an image type suitable for the displacement detection target is determined, the designation information includes information indicating the displacement detection target. Note that when the entire target image is the area that is to be subjected to displacement extraction and an image type suitable for the purpose is not determined, the designation information does not need to be used. Details of the designation information will be described later.

[0016] Furthermore, the user selects a sensor image to be used as a target image from among the sensor images stored in the image storage unit 22. The input receiving unit 23 receives an input of the selection result of the sensor image to be used as the target image, and transmits the sensor image corresponding to the selection result to the displacement extraction device 3 as the target image. Similarly, the input receiving unit 23 receives an input of the selection result of the sensor image to be used as the reference image, and transmits the sensor image corresponding to the selection result to the displacement extraction device 3 as the reference image. The image type of the target image and the image type of the reference image may be the same or different. For example, both the target image and the reference image may be SAR images, both the target image and the reference image may be optical images, or one of the target image and the reference image may be an optical image and the other a SAR image.

[0017] The transmitting / receiving unit 24 transmits the designation information, the target image, and the reference image to the displacement extraction device 3. As described above, the designation information does not have to be used, in which case the transmitting / receiving unit 24 transmits the target image and the reference image to the displacement extraction device 3. The transmitting / receiving unit 24 also receives the displacement extraction result from the displacement extraction device 3 and outputs it to the display unit 25. The display unit 25 displays the displacement extraction result.

[0018] Although an example in which the target image and the reference image are stored in the image storage unit 22 is described here, the method by which the displacement extraction device 3 acquires the target image and the reference image is not limited to this. For example, a user may have acquired a right to use an image, and the user terminal 2 may use this right to acquire the target image and the reference image from the image providing system 4 and transmit them to the displacement extraction device 3. Alternatively, a user may provide the target image and the reference image via a recording medium, and the displacement extraction device 3 may acquire the target image and the reference image by reading them from the recording medium. Alternatively, the target image and the reference image may be provided to the displacement extraction device 3 in different ways. For example, the target image may be provided to the displacement extraction device 3 via a recording medium, and the user terminal 2 may acquire the reference image from the image providing system 4 and transmit it to the displacement extraction device 3. This combination is merely an example, and the combination of the methods for providing the target image and the reference image is not limited to this example.

[0019] When it becomes necessary to obtain displacement extraction results, even if a user can obtain the latest SAR image of the area from which the displacement is to be extracted, there may be no SAR image of the same area observed as a reference image before the displacement occurred, and only an optical image may be obtained. Conversely, there may be a case where the latest optical image of the area from which the displacement is to be extracted is obtained, but no optical image of the same area observed as a reference image before the displacement occurred, and only a SAR image may be obtained. Note that displacement extraction is performed, for example, to grasp the situation of a disaster, take disaster prevention measures, monitor specific locations, etc., but the uses of displacement extraction are not limited to these examples.

[0020] For example, in the event of a small-scale disaster, observation using an optical sensor from an unmanned aerial vehicle (hereinafter also referred to as ground observation) may be performed near the disaster site. In the event of a large-scale disaster, emergency observation using a satellite may be performed. However, in the event of a small-scale disaster, observation using an optical sensor mounted on an unmanned aerial vehicle is more likely to be performed to facilitate observation. In this case, a sensor image taken before the disaster occurs is used as the reference image, but an optical image of the corresponding area may not necessarily be obtained; in some cases, only an SAR image of the corresponding area may be obtained. In satellite observation, optical images cannot be observed at night using visible band observations, and cannot be observed below clouds even when there are clouds. However, SAR images can be observed at night and even when there are clouds. For this reason, SAR generally has more opportunities for observation than optical sensors. When the target image is an optical image and the reference image is an SAR image, it is difficult to extract displacement as is. However, in the displacement extraction device 3 of this embodiment, when the image types of the target image and the reference image are different, one sensor image is converted to an image of the other image type, and displacement is extracted using the converted image and one sensor image. As a result, in this embodiment, it is possible to increase the possibility of providing a user with a highly accurate displacement extraction result. Also, there are cases where a user owns a SAR image as a target image and an optical image that can be used as a reference image. In such a case, by providing the displacement extraction device 3 with the SAR image that is the target image and the optical image that is the reference image that the user owns, the user can obtain a displacement extraction result without separately preparing a reference image that is a SAR image.

[0021] As shown in FIG. 1, the displacement extraction device 3 includes a specified information acquisition unit 31, an image type determination unit 32, a target image acquisition unit 33, a target image cropping unit 34, a reference image acquisition unit 35, a reference image cropping unit 36, a conversion necessity determination unit 37, an image conversion unit 38, a position alignment unit 39, a displacement extraction unit 40, and an output unit 41.

[0022] The designation information acquisition unit 31 acquires the designation information by receiving it from the user terminal 2 and outputs it to the image type determination unit 32, the target image cropping unit 34, and the reference image cropping unit 36. While an example in which the designation information is transmitted from the user terminal 2 is described here, the designation information acquisition unit 31 may acquire the designation information by receiving input from an operator or the like as an input means. As described above, the designation information includes area information indicating a target area, which is an area of ​​the target image from which displacement is extracted. The area information may be indicated by information indicating the positions of the four corners of the target area, or information indicating the center position and size of the target area, or other information. The information indicating the position may be latitude and longitude or other coordinate values. Furthermore, as described above, if the entire target image is an area from which displacement is extracted and an image type appropriate for the purpose is not determined, the designation information may not be used. In this case, the displacement extraction device 3 does not need to include the designation information acquisition unit 31.

[0023] The image type determination unit 32 determines an image type suitable for the displacement detection target using information indicating the displacement detection target included in the specification information received from the specification information acquisition unit 31, and outputs the determined result to the conversion necessity determination unit 37. The determined result of the image type determination unit 32 is used to determine whether to match the image type, SAR image or optical image, when the image types of the target image and the reference image are different.

[0024] The target image acquisition unit 33 acquires a target image, which is an image of a target region that is the target of displacement extraction. For example, the target image acquisition unit 33 acquires the target image by receiving the target image from the user terminal 2, and outputs the acquired target image to the target image cropping unit 34. As described above, the target image acquisition unit 33 may acquire the target image by reading it from a recording medium, or may acquire the target image by receiving a target image that the user has the right to use from the image providing system 4 via the user terminal 2.

[0025] The target image cropping unit 34 uses the area information included in the specification information received from the specification information acquisition unit 31 to crop an image corresponding to the target area for displacement extraction from the target image received from the target image acquisition unit 33, and outputs the cropped image to the conversion necessity determination unit 37. It is assumed that the target image includes information indicating the position of the imaging area corresponding to the target image.

[0026] The reference image acquisition unit 35 acquires a reference image, which is an image of the target area and has a different capture date and time from the target image. For example, the reference image acquisition unit 35 acquires the reference image by receiving the reference image from the user terminal 2, and outputs the acquired reference image to the reference image cropping unit 36. As described above, the reference image acquisition unit 35 may acquire the reference image by reading it from a recording medium, or may acquire the reference image by receiving a reference image, for which the user has the right to use, from the image providing system 4 via the user terminal 2.

[0027] The reference image cropping unit 36 ​​uses the region information included in the specification information received from the specification information acquisition unit 31 to crop an image corresponding to the target region for displacement extraction from the reference image received from the reference image acquisition unit 35, and outputs the cropped image to the conversion necessity determination unit 37. It is assumed that the reference image includes information indicating the position of the imaging region corresponding to the reference image.

[0028] Note that if the entire target image acquired by the target image acquisition unit 33 is to be the target of displacement extraction, the target image clipping unit 34 may not be provided. In this case, the reference image clipping unit 36 ​​clips out an image of a portion corresponding to the imaging area of ​​the target image from the reference image. Alternatively, the entire reference image acquired by the reference image acquisition unit 35 may be the target of displacement extraction, in which case the reference image clipping unit 36 ​​may not be provided. In this case, the target image clipping unit 34 clips out an image of a portion corresponding to the imaging area of ​​the reference image from the target image.

[0029] Alternatively, the specification information may not specify region information, and the target image cropping unit 34 and the reference image cropping unit 36 ​​may each crop an image of a portion corresponding to an overlapping imaging region between the reference image and the target image. Furthermore, if the size of the target image is larger than the reference image, the reference image acquisition unit 35 may acquire multiple reference images corresponding to one target image. For example, if joining three reference images results in an image corresponding to the imaging region of the target image, the target image may be divided into three reference images. In this case, three pairs of target image and reference image are generated.

[0030] The conversion necessity determination unit 37 determines whether the image type of the cropped target image received from the target image cropping unit 34 is the same as the image type of the cropped reference image received from the reference image cropping unit 36. For example, each sensor image may be provided with information for determining the image type, such as information indicating the sensor that captured the sensor image or information indicating the image type itself, and the conversion necessity determination unit 37 may determine the image type based on this information, or the conversion necessity determination unit 37 may determine the image type based on the format of the sensor image. As described above, cropping of at least one of the target image and the reference image may not be performed. In this case, an image that is not cropped is input to the conversion necessity determination unit 37. Since cropping is not required, the cropped target image input to the conversion necessity determination unit 37 will hereinafter be referred to as the target image, and the cropped reference image input to the conversion necessity determination unit 37 will hereinafter be referred to as the reference image.

[0031] If the image type of the target image and the image type of the reference image are the same, the conversion necessity determination unit 37 determines that image conversion is unnecessary and outputs the target image and the reference image to the alignment unit 39. If the image type of the target image and the image type of the reference image are not the same, the conversion necessity determination unit 37 selects a sensor image to be converted based on the image type determination result notified from the image type determination unit 32, outputs the selected sensor image to the image conversion unit 38, and outputs sensor images not to be converted to the alignment unit 39. Specifically, if the image type determination result is an SAR image, the conversion necessity determination unit 37 selects the sensor image, which is an optical image, from the target image and the reference image as the sensor image to be converted and outputs it to the image conversion unit 38, and instructs the image conversion unit 38 to convert the optical image into a SAR image. If the image type is determined to be an optical image, the sensor image that is an SAR image from among the target image and the reference image is selected as the conversion target and output to the image conversion unit 38, and the image conversion unit 38 is instructed to convert the SAR image into an optical image.

[0032] When the image type of the reference image and the image type of the target image are different, the image conversion unit 38 generates a converted image by converting a first image, which is one of the reference image and the target image, into an image of the image type of the second image, which is the other image. Note that the image type determination unit 32 described above corresponds to determining the type of this second image. Specifically, the image conversion unit 38 converts the image type of the sensor image received from the conversion necessity determination unit 37 based on an instruction from the conversion necessity determination unit 37. Specifically, when the image conversion unit 38 receives an SAR image from the conversion necessity determination unit 37, it converts the SAR image into an optical image based on the instruction from the conversion necessity determination unit 37 to generate a pseudo-optical image, and outputs the generated pseudo-optical image to the alignment unit 39. Furthermore, when the image conversion unit 38 receives an optical image from the conversion necessity determination unit 37, it converts the optical image into a SAR image based on the instruction from the conversion necessity determination unit 37 to generate a pseudo-SAR image, and outputs the generated pseudo-SAR image to the alignment unit 39. Hereinafter, the pseudo-optical image and the pseudo-SAR image will also be referred to as transformed images.

[0033] The alignment unit 39 aligns the set of input sensor images and outputs the aligned set of sensor images to the displacement extraction unit 40. When image conversion is not performed, the set of images input to the alignment unit 39 is a set of a target image received from the conversion necessity determination unit 37 and a reference image received from the conversion necessity determination unit 37. When image conversion is performed, the set of images input to the alignment unit 39 is a set of a target image received from the conversion necessity determination unit 37 and a reference image that is a converted image received from the image conversion unit 38, or a set of a reference image received from the conversion necessity determination unit 37 and a target image that is a converted image received from the image conversion unit 38.

[0034] When the image type of the reference image and the image type of the target image are different, the displacement extraction unit 40 extracts the displacement by comparing the second image that has not been subjected to image conversion by the image conversion unit 38 with the converted image that has been converted by the image conversion unit 38. When the image type of the reference image and the image type of the target image are the same, the displacement extraction unit 40 extracts the displacement by comparing the target image with the reference image.

[0035] Specifically, the displacement extraction unit 40 uses a pair of images input from the alignment unit 39 to extract locations where displacement has occurred between the pair of images and output the extraction results to the output unit 41. Any method for extracting the displacement may be used. For example, the difference in pixel value for each pixel may be calculated, and pixels where the difference is equal to or greater than a threshold value may be extracted as locations where displacement has occurred. Alternatively, instead of each pixel, a group of consecutive pixels may be used as a unit, and locations where the difference in the average values ​​of the pixels is equal to or greater than a threshold value may be extracted as locations where displacement has occurred. The extraction results may be data for each pixel, such as assigning a pixel value of 1 to pixels determined to have displacement and a pixel value of 0 to pixels determined not to have displacement. Alternatively, the extraction results may be information indicating the position of a pixel determined to have displacement and the amount of displacement (difference). Furthermore, the displacement extraction unit 40 may perform some kind of image processing on the pair of images before calculating the difference. Furthermore, the extraction results may be data in which a stepped numerical value corresponding to the magnitude of displacement is assigned to each pixel as a pixel value. For example, the pixel value difference itself may be assigned to a pixel determined to have a displacement, or a pixel value may be assigned in stages, such as 1 if the difference is equal to or greater than a first threshold, or 2 if the difference is equal to or greater than a second threshold that is greater than the first threshold, etc. Any method may be used to extract the displacement, and is not limited to the above-mentioned example.

[0036] The output unit 41 outputs the extraction result by transmitting the extraction result received from the displacement extraction unit 40 to the user terminal 2. The output unit 41 may also output the extraction result by displaying the extraction result. If the extraction result is expressed as a pixel value for each pixel, the extraction result indicating the location where the displacement has occurred corresponds to an image, so the user terminal 2 and the output unit 41 can display the displacement extraction result as an image. The displacement extraction unit 40 may also generate an image in which the displacement extraction result is superimposed on the target image or reference image, and the output unit 41 may also output the image.

[0037] Next, the operation of this embodiment will be described. Fig. 2 is a flowchart showing an example of displacement extraction processing in the displacement extraction device 3 of this embodiment. First, the displacement extraction device 3 acquires designation information (step S1). In detail, the designation information acquisition unit 31 acquires the designation information and outputs the acquired designation information to the image type determination unit 32, the target image cutout unit 34, and the reference image cutout unit 36.

[0038] Next, the displacement extraction device 3 determines an image type (step S2). Specifically, the image type determination unit 32 determines an image type corresponding to the displacement detection target (type of displacement detection target) using information indicating the displacement detection target. For example, the image type determination unit 32 stores correspondence information indicating the correspondence between the displacement detection target and the corresponding image type, and determines an image type appropriate for the displacement detection target by extracting an image type corresponding to the information indicating the displacement extraction target from the correspondence information. FIG. 3 is a diagram showing an example of image type correspondence information according to this embodiment. For example, when the displacement extraction target is water, such as a flooded area during a flood, an optical sensor may observe the same water in different colors. Specifically, in cases such as muddy water, it may be difficult to determine the flooded area, and SAR images may be more appropriate. In this way, determining an image type appropriate for the displacement extraction target according to the characteristics of each sensor can improve the accuracy of displacement extraction.

[0039] The image type determination unit 32 may also determine an image type suitable for a displacement extraction target using machine learning. For example, for each displacement extraction target, both displacement extraction using SAR images and displacement extraction using optical images are performed to obtain the actual displacement, and then which of the displacements between the SAR images and the optical images had higher accuracy is obtained as ground truth data. Then, in the learning phase, the image type determination unit 32 or another learning device (not shown) generates a trained model using multiple pieces of training data consisting of information indicating the displacement extraction target and ground truth data, and stores the generated trained model in a storage unit (not shown) within the image type determination unit 32. Then, in the inference phase, i.e., when determining the image type, the image type determination unit 32 inputs information indicating the displacement extraction target into the trained model to obtain an inference result indicating whether the SAR image or the optical image is suitable, and sets the inference result as the image type determination result.

[0040] Here, an example is described in which the image type is determined based on information indicating the displacement extraction target. However, this is not limiting, and an image type suitable for the displacement detection target may be specified by the user. For example, the specified information may include information indicating an image type suitable for the displacement detection target instead of information indicating the displacement extraction target. Alternatively, the image type determination unit 32 may function as an input unit, accept input of an image type, and set the accepted image type as the image type suitable for the displacement detection target. Alternatively, if the image type of the target image differs from the image type of the reference image, it may be determined in advance which image type to use, such as performing displacement extraction using the image type of the target image or the image type of the reference image.

[0041] Returning to the explanation of Fig. 2, next, the displacement extraction device 3 acquires a target image and a reference image (step S3). In detail, the target image acquisition unit 33 acquires the target image from the user terminal 2 and outputs the acquired target image to the target image clipping unit 34, and the reference image acquisition unit 35 acquires the reference image from the user terminal 2 and outputs the acquired reference image to the reference image clipping unit 36. Note that, although step S3 is described after step S2 in Fig. 2, step S3 may be performed simultaneously with step S1, or steps S1 and S2 may be performed after step S3.

[0042] Next, the displacement extraction device 3 cuts out the target image and the reference image (step S4). In detail, the target image cutout unit 34 cuts out the target image and outputs the cut-out target image to the conversion necessity determination unit 37, and the reference image cutout unit 36 ​​cuts out the reference image and outputs the cut-out reference image to the conversion necessity determination unit 37. As described above, cutout does not have to be performed on at least one of the target image and the reference image.

[0043] Next, the displacement extraction device 3 determines whether the target image and the reference image are of the same type (step S5). Specifically, the conversion necessity determination unit 37 determines whether the image type of the cut-out target image (the target image that is not cut out when cut-out is not performed) is the same as the image type of the cut-out reference image (the reference image that is not cut out when cut-out is not performed).

[0044] If the target image and the reference image are not of the same type (No in step S5), that is, if the image type of the target image and the image type of the reference image are different, the displacement extraction device 3 performs image conversion (step S6). In detail, the conversion necessity determination unit 37 selects a sensor image to be subjected to image conversion from the target image and the reference image based on the determination result of the image type determination unit 32, and outputs the selected sensor image to the image conversion unit 38. The image conversion unit 38 performs image conversion on the input sensor image to generate a converted image.

[0045] The image conversion in the image conversion unit 38 may be performed by any method, for example, a method using machine learning as described below. Note that the image conversion method in the image conversion unit 38 may be a method in which conversion is performed by performing various processes such as orthorectification, and is not limited to a method using machine learning.

[0046] When image conversion in the image conversion unit 38 is performed using a method using machine learning, for example, a learning device generates a first trained model, which is an inference model for converting SAR images into optical images, and a second trained model, which is an inference model for converting optical images into SAR images. Then, the image conversion unit 38 converts SAR images into optical images using the first trained model, and converts optical images into SAR images using the second trained model. Note that the learning device may be provided within the displacement extraction device 3. Machine learning can be performed, for example, by supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, etc. The following mainly describes the case where supervised learning is performed.

[0047] The following description will first take as an example a case where an SAR image is converted into an optical image. FIG. 4 is a diagram showing an example of the configuration of a learning device 5 according to this embodiment. As shown in FIG. 4, the learning device 5 includes an image acquisition unit 51, a learning data generation unit 52, a model generation unit 53, and a trained model storage unit 54. The image acquisition unit 51 acquires an SAR image and an optical image, and outputs the acquired SAR image and optical image to the learning data generation unit 52. The learning data generation unit 52 regards a pair of an SAR image and an optical image as one set of learning data, and generates and outputs multiple sets of learning data. When learning is performed by supervised learning, each of the multiple sets of learning data is composed of a pair of an SAR image as input data and an optical image as correct output data. The optical image as correct output data refers to an optical image that should be output as a conversion result when an SAR image is input as input data.

[0048] The above pairs are generated from SAR images and optical images obtained by capturing images at the same time. Capturing images at the same time means capturing images simultaneously or with a time difference so short that changes in the images are negligible. These SAR images and optical images may be acquired by satellite observation or by terrestrial observation. The SAR images and optical images used to generate the learning data may or may not have the same imaging area. If they do not match, they may each contain at least a common portion (a common subject). If they do not match, the learning data generation unit 52 creates a pair of SAR images and optical images from such SAR images and optical images that have the same imaging area, and uses this pair as a set of learning data.

[0049] Each of the SAR images and optical images constituting the training data must be data cut out into patches of a predetermined size. The patch size is defined, for example, by the number of pixels in the vertical and horizontal directions. Here, the patch sizes of the SAR images and optical images may be the same or different.

[0050] The model generation unit 53 generates, by machine learning, a first trained model that converts a SAR image into an optical image, based on the multiple sets of training data output from the training data generation unit 52. Converting a SAR image into an optical image means inferring, from the SAR image, an optical image that corresponds to the SAR image.

[0051] When machine learning is performed by supervised learning, the model generation unit 53 performs learning so that when an SAR image from each set of learning data is input to the first trained model, the converted optical image output from the first trained model is sufficiently close to the correct optical image.

[0052] The first trained model is configured, for example, by a neural network. FIG. 5 is a simplified diagram showing an example of a neural network. The neural network shown in FIG. 5 has an input layer 501, an intermediate layer (hidden layer) 502, and an output layer 503. In the example shown in FIG. 5, the number of intermediate layers 502 is one, but the number of intermediate layers 502 may be two or more. The number of neurons in each layer is also not limited to the example shown in FIG. 5.

[0053] Pixel values ​​of the SAR image are input to neurons P in the input layer 501. Neurons P in the input layer 501 output the input as is. Neurons P in the output layer 503 output pixel values ​​of the optical image generated by the conversion.

[0054] Each of the neurons in the hidden layer 502 and the output layer 503 performs an operation expressed by the following equation (1) on a plurality of inputs. y=s(w1×x1+w2×x2+····+w N ×x N +b) …(1)

[0055] In equation (1), N is the number of inputs to neuron P, which is not necessarily the same between neurons. N is the input data of neuron P, w1~w N is input x1~x N a is the weight for a, and b is the bias. The weights and biases are determined through learning. Hereinafter, the weights and biases are collectively referred to as parameters. The function s(a) is the activation function. The activation function is not necessarily the same between neurons.

[0056] As shown in Fig. 4, the model generation unit 53 includes a pixel value comparison unit 531, a parameter adjustment unit 532, and a memory 533. The memory 533 holds an inference model under training. The inference model under training stored in the memory 533 is updated as training progresses. The SAR image of each set of training data is input to the inference model under training, and the optical image as the correct answer is input to the pixel value comparison unit 531. When an SAR image is input to the inference model under training, a converted optical image that is an optical image converted by the inference model under training is output.

[0057] The pixel value comparison unit 531 compares the converted optical image with an optical image serving as the correct data in the learning data, and calculates a loss function based on the comparison result. For example, the loss function is the absolute value or the sum of squares of the difference in pixel value for each pixel between the converted optical image and the optical image.

[0058] The parameter adjustment unit 532 adjusts the parameters of the inference model being trained based on the loss function. The parameter adjustment is performed so that the loss function is reduced. The parameter adjustment can be performed using, for example, backpropagation and gradient descent.

[0059] The parameter adjustment and inference (image transformation) using the adjusted parameters are repeated, and when the loss function becomes less than a predetermined threshold, the learning ends and a trained inference model is obtained. The trained first trained model is stored in the trained model storage unit 54.

[0060] Note that instead of temporarily storing the inference model under training in the memory 533 in the model generation unit 53, the inference model under training may be stored in the trained model storage unit 54. In this case, the inference model stored at the time when training is completed becomes the first trained model.

[0061] Furthermore, in the above example, the learning device 5 newly generates the first trained model, but the first trained model that has been generated by learning may be retrained.

[0062] 6 is a diagram showing an example of the configuration of the image conversion unit 38 according to the present embodiment when image conversion is performed by machine learning. The image conversion unit 38 shown in FIG. 6 includes an inference unit 381 and a trained model storage unit 382.

[0063] The trained model storage unit 382 stores a first trained model generated by the learning device 5. The inference unit 381 converts an input SAR image into a pseudo-optical image using the first trained model stored in the trained model storage unit 382. That is, the inference unit 381 inputs the SAR image into the first trained model, thereby inferring and outputting a pseudo-optical image corresponding to the SAR image.

[0064] When converting an optical image into a SAR image, the above-described process of swapping the SAR image and the optical image is performed. That is, the learning device 5 generates a second trained model by performing the above-described process of swapping the SAR image and the optical image. The trained model storage unit 382 of the image conversion unit 38 also stores the second trained model. As a result, when converting an optical image into a SAR image, the inference unit 381 inputs the optical image into the second trained model, thereby inferring and outputting a pseudo-SAR image corresponding to the optical image.

[0065] Furthermore, while optical images change in a variety of ways depending on the time of image capture, weather conditions, etc., SAR images change less depending on the time of image capture, weather conditions, etc. Therefore, compared to converting SAR images to optical images, converting optical images to SAR images has less ill-posedness during image conversion, and there is a high possibility that an image closer to the correct answer can be generated.

[0066] In the above example, the first trained model and the second trained model are generated by supervised learning. Instead of supervised learning, the first trained model and the second trained model can also be generated by unsupervised learning, semi-supervised learning, reinforcement learning, or the like. Even in the case of unsupervised learning, SAR images and optical images are required as multiple sets of learning data, but in this case, the images do not need to include the same subject. In other words, they do not need to be paired data.

[0067] In addition, deep learning, which learns to extract feature points and feature quantities, can be used to generate the first trained model and the second trained model, or other known methods such as genetic programming, functional logic programming, and support vector machines can also be used.

[0068] Furthermore, in the above-described learning phase, learning may be performed by including not only the sensor image but also the distribution of difference values ​​between the correct sensor image and the generated converted image as the correct data in the learning data as a probability map. That is, learning may be performed by including the difference values ​​between the correct sensor image and the generated converted image as the accuracy of the image conversion in the correct data. In this way, in the inference phase, the accuracy map can be obtained along with the converted image.

[0069] Returning to the explanation of Fig. 2, the displacement extraction device 3 performs registration (step S7). More specifically, the registration unit 39 performs registration between a pair of input images.

[0070] The alignment by the alignment unit 39 may be performed by any method, but for example, it can be performed by the following method.

[0071] The alignment unit 39 detects misalignment between the target image and the reference image and outputs misalignment information indicating the detected misalignment. The alignment unit 39 can use, for example, feature point matching or block matching to detect misalignment. In feature point matching, feature points of one image are extracted, and corresponding feature points of the other image that correspond to the feature points of the other image are identified, and the difference in position between the corresponding feature points is detected. Feature point matching may be, for example, matching using KAZE (meaning "wind" in Japanese), AKAZE (AcceleratedKAZE), or SURF (Speed-UpRobustFeatures) features. In block matching, an image is divided into multiple block regions, and for each block region, corresponding blocks of one image are identified based on the similarity, and the difference in position between the corresponding blocks is detected.

[0072] The position adjustment unit 39 corrects the position of the target image based on the positional deviation information to generate a correction target image. For example, the position adjustment in the position adjustment unit 39 is performed so that feature points or blocks determined to correspond in the two images coincide with each other or become closer to each other.

[0073] In the above example, the target image is corrected for alignment to generate the correction target image, but instead, the reference image may be corrected for alignment to generate the corrected reference image. Also, correction may be performed on both the target image and the reference image. When correcting both the target image and the reference image, the amount of correction for each image can be made smaller.

[0074] Next, the displacement extraction device 3 performs displacement extraction (step S8). Specifically, the displacement extraction unit 40 extracts the displacement as described above using a pair of images after alignment. If an accuracy map is acquired along with the converted image, the accuracy map may be used for displacement extraction. Specifically, in Example 1, when extracting the displacement, processing may be added such that displacement is not extracted for areas where the accuracy is lower than a predetermined value. In Example 2, the displacement may be extracted, information indicating the displacement is aligned with the accuracy map, and the information indicating the displacement and information indicating the accuracy may be displayed in association with each other. The information indicating the accuracy in Example 2 may be displayed in a heat map, with low accuracy indicated by a light color and high accuracy indicated by a dark color. This makes it possible to prevent erroneous displacement extraction due to conversion errors occurring during conversion. Next, the displacement extraction device 3 outputs the extraction result (step S9) and terminates the process. Specifically, for example, the output unit 41 may output the extraction result by transmitting the displacement extraction result to the user terminal 2, or the output unit 41 may output the extraction result by displaying the displacement extraction result. Furthermore, when an accuracy map is acquired together with the converted image, the output unit 41 outputs the extraction result in association with the accuracy map. The output unit 41 may, for example, transmit a map showing the displacement extraction results as a distribution and the displacement extraction result accuracy map to the user terminal 2, or may display the map showing the displacement extraction results as a distribution and the displacement extraction result accuracy map in association with each other. Specifically, the output unit 41 may, for example, superimpose the map showing the displacement extraction results as a distribution and the displacement extraction result accuracy map, for example, and indicate low accuracy with a light color and high accuracy with a dark color. Similarly, the user terminal 2 may display the map showing the displacement extraction results as a distribution and the displacement extraction result accuracy map in association with each other.

[0075] If the answer is Yes in step S5, the displacement extraction device 3 advances the process to step S7. That is, if the answer is Yes in step S5, the displacement extraction device 3 performs the processes from step S7 onwards without performing image conversion.

[0076] Next, the hardware configuration of the displacement extraction device 3 of this embodiment will be described. The displacement extraction device 3 of this embodiment functions as the displacement extraction device 3 by executing a computer program on the computer system, which is a computer program describing the processing to be performed by the displacement extraction device 3. FIG. 7 is a diagram showing an example of the configuration of a computer system that realizes the displacement extraction device 3 of this embodiment. As shown in FIG. 7, this computer system includes a control unit 101, an input unit 102, a storage unit 103, a display unit 104, a communication unit 105, and an output unit 106, which are connected via a system bus 107.

[0077] In FIG. 7 , the control unit 101 is a processor such as a CPU (Central Processing Unit) and executes a program describing the processing of the displacement extraction device 3 of this embodiment. Note that a portion of the control unit 101 may be realized by dedicated hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). The input unit 102 is composed of, for example, a keyboard, a mouse, etc., and is used by a user of the computer system to input various information. The memory unit 103 includes various memories such as RAM (Random Access Memory) and ROM (Read Only Memory) and a storage device such as a hard disk, and stores programs to be executed by the control unit 101, necessary data obtained during processing, etc. The memory unit 103 is also used as a temporary storage area for programs. The display unit 104 is composed of a display, an LCD (Liquid Crystal Display Panel), etc., and displays various screens to the user of the computer system. The communication unit 105 is a receiver and transmitter that performs communication processing. The output unit 106 is a printer, a speaker, etc. Note that FIG. 7 is just an example, and the configuration of the computer system is not limited to the example of FIG.

[0078] Here, an example of the operation of the computer system until the program of this embodiment is ready to be executed will be described. In the computer system having the above-mentioned configuration, for example, a computer program is installed in the storage unit 103 from a CD-ROM or DVD-ROM inserted in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown). Then, when the program is executed, the program read from the storage unit 103 is stored in the main storage area of ​​the storage unit 103. In this state, the control unit 101 executes processing as the displacement extraction device 3 of this embodiment in accordance with the program stored in the storage unit 103.

[0079] In the above description, a program describing the processing in the displacement extraction device 3 is provided on a CD-ROM or DVD-ROM as a recording medium, but this is not limiting. Depending on the configuration of the computer system, the capacity of the program to be provided, etc., it is also possible to use a program provided via a transmission medium such as the Internet via the communication unit 105.

[0080] The program of this embodiment executes, for example, the steps of acquiring a target image, which is an image of a target area from which displacement is to be extracted; acquiring a reference image, which is an image of the target area and has a different image capture date and time from the target image; generating a converted image by converting a first image, which is one of the reference image and the target image, into an image of the image type of the other image, which is a second image, if the image type of the reference image and the target image is different; extracting displacement by comparing the second image with the converted image; and outputting the displacement extraction result.

[0081] The image type determination unit 32, target image cropping unit 34, reference image cropping unit 36, conversion necessity determination unit 37, image conversion unit 38, alignment unit 39, and displacement extraction unit 40 shown in FIG. 1 are realized by the control unit 101 shown in FIG. 7 executing a computer program stored in the storage unit 103 shown in FIG. 7. The storage unit 103 shown in FIG. 7 is also used to realize the image type determination unit 32, target image cropping unit 34, reference image cropping unit 36, conversion necessity determination unit 37, image conversion unit 38, alignment unit 39, and displacement extraction unit 40 shown in FIG. 1. The specified information acquisition unit 31, target image acquisition unit 33, reference image acquisition unit 35, and output unit 41 shown in FIG. 1 are realized by the communication unit 105 shown in FIG. 7. Furthermore, when the specified information acquisition unit 31 shown in FIG. 1 accepts input, the specified information acquisition unit 31 is realized by the input unit 102 shown in FIG. 7. 1 acquire sensor images by reading a recording medium, the target image acquisition unit 33 and the reference image acquisition unit 35 are realized by a recording medium reader or the like connected to the computer system shown in Fig. 7. The displacement extraction device 3 may also be realized by a plurality of computer systems. For example, the displacement extraction device 3 may be realized by a cloud computer system.

[0082] Similarly, the user terminal 2 and the image providing system 4 are realized by a computer system having the configuration shown in FIG.

[0083] As described above, the displacement extraction device 3 of this embodiment acquires a target image, which is a sensor image capturing a target area for displacement extraction, and a reference image, which is a sensor image used as a reference. If the image types of the target image and the reference image are different, one of the sensor images is converted to an image of the other image type, and displacement is extracted using the image obtained by conversion and the one sensor image. This increases the likelihood that a highly accurate displacement extraction result will be provided to the user. Furthermore, since the image types of the target image and the reference image may be different in this embodiment, the user can obtain displacement extraction results by effectively utilizing sensor images that they own or have the right to use.

[0084] Embodiment 2 FIG. 8 is a diagram showing an example of the configuration of a displacement extraction system according to the second embodiment. The displacement extraction system 1a of this embodiment includes a displacement extraction device 3a and a user terminal 2a. The user terminal 2a is obtained by removing the image acquisition unit 21 and the image storage unit 22 from the user terminal 2 of the first embodiment. The displacement extraction device 3a adds an image request unit 42 and an image confirmation unit 43 to the displacement extraction device 3 of the first embodiment, and includes a specified information acquisition unit 31a instead of the specified information acquisition unit 31. Components having the same functions as those of the first embodiment are assigned the same reference numerals as those of the first embodiment, and redundant explanations will be omitted. Below, differences from the first embodiment will be mainly explained.

[0085] In the first embodiment, an example was described in which displacement is extracted using a target image and a reference image that the user owns or has the right to use, but the displacement extraction device 3a in this embodiment acquires a target image and a reference image that meet the user's request based on specification information that indicates the user's request from the image providing system 4. Then, the displacement extraction device 3a extracts displacement in the same manner as in the first embodiment, using the target image and the reference image acquired from the image providing system 4.

[0086] In this embodiment, the specification information includes target date and time information, which is information regarding the date and time of capture of each of the target image and the reference image, in addition to the region information and information indicating the displacement detection target described in the first embodiment. The information regarding the date and time of capture may be information indicating the date and time of capture or information indicating the time period of capture. The time period of capture has a time range, in other words, a predetermined allowable range. The specification of the target date and time information may be selected from multiple specification methods (target date and time specification methods). The specification information acquisition unit 31a acquires the specification information from the user terminal 2a and outputs the specification information to the image request unit 42. The image request unit 42 uses the target date and time information and region information in the specification information to search for sensor images from the image providing system 4 and output the searched sensor images to the image confirmation unit 43. The sensor images provided by the image providing system 4 may be obtained by satellite observation or by ground observation using an aircraft, etc. Note that while FIG. 8 illustrates one image providing system 4, the image providing system 4 may be a combination of multiple systems. For example, the image providing system 4 may be divided into a system that provides SAR images and a system that provides optical images. Furthermore, the image providing system 4 may be an external system, or a system managed by the operator of the displacement extraction device 3a. In the latter case, the image providing system 4 may acquire and store SAR images and optical images from the external system in advance.

[0087] FIG. 9 is a diagram showing an example of a target date and time designation method when the target date and time information according to the present embodiment indicates the date and time of photography. In the example shown in FIG. 9, the user can select one of three target date and time designation methods: point designation, reference date designation, and range designation. Point designation is a method for designating the date and time of a single point at which a sensor image was captured. In point designation, the user designates the date and time to be designated and an allowable range, which is an allowable range for deviation from the date and time. Reference date designation is a method for designating whether the date and time is before or after the reference date. In reference date designation, the user designates the date and time to be the reference date, selects whether it is before or after the reference date (before or after selection), and selects whether to prioritize a closer date and time or a season, weather, or the like (priority item selection). Note that if the closer date and time is prioritized in the priority item selection, a sensor image with a date and time closest to the reference date that is before (or after) the reference date is acquired. Furthermore, if the season is prioritized in the priority item selection, the season is also designated, and sensor images for the specified season are acquired. Furthermore, if weather is selected as a priority in the priority item selection, the weather is also specified, and a sensor image of the specified weather is acquired. Range specification is a method of specifying a range, in which the user specifies the start date and time of the range, the end date and time of the range, and the priority conditions. Furthermore, if season priority is specified, the reference image may be specified to be in the same season as the target image. In this case, the image request unit 42 acquires an image captured in the same season as the target image as the reference image. In this embodiment, an example of a target date and time specification method when the target date and time information is information indicating the time of shooting is to specify October 2021, spring 2021, etc. Priority item selection in this case is the same as when the target date and time information is information indicating the date and time of shooting.

[0088] For example, when a user wants to specify a relatively narrow range of date and time for a target image, such as when the user wants to check the changes immediately after a specific event occurs, the user specifies the date and time for the target image by specifying a point. On the other hand, in this case, the reference image only needs to be taken before the event occurs, and there are few time constraints. Therefore, for example, the user may specify a relatively wide range for the date and time of the reference image by specifying a range. Alternatively, when the user wants to check the changes before and after a specific event, the user may specify a date after the reference date for the target image by specifying a reference date, and a date before the reference date for the reference image by specifying a reference date. The above-described target date and time specification methods are merely examples, and the number and content of target date and time specification methods and the items specified in each specification method are not limited to the example shown in FIG. 9.

[0089] The target date and time designation method is selected, for example, by the displacement extraction device 3a displaying a screen on the user terminal 2a that accepts the selection of the target date and time designation method, and the input accepting unit 23 of the user terminal 2a accepting the input of the selection result from the user. When the target date and time designation method is selected, a screen for inputting each designation item corresponding to the selected designation method is displayed on the user terminal 2a. Note that, although an example will be described here in which the user terminal 2a accepts the designation information and transmits it to the displacement extraction device 3a, as described above, the designation information may also be input directly to the displacement extraction device 3a.

[0090] The image confirmation unit 43 confirms the quality of the sensor image received from the image request unit 42. For example, if information that can determine the quality is attached to the sensor image or is notified separately from the sensor image, the quality may be confirmed using that information. Alternatively, the quality may be confirmed by determining whether the sensor image satisfies a predetermined condition for quality determination, such as determining whether the sensor image is significantly affected by the atmosphere based on the season, or determining whether the sensor image is cloudy if it is an optical image. Furthermore, if the sensor image was captured before a predetermined period, the quality may be determined to be insufficient. If the quality of the sensor image does not satisfy the condition, the image confirmation unit 43 instructs the image request unit 42 to search for the sensor image again, and the image request unit 42 again searches for the sensor image from the image providing system 4 using the target date and time information and area information in the specified information.

[0091] FIG. 10 is a flowchart showing an example of image acquisition processing according to this embodiment. In this embodiment, the processing shown in FIG. 10 is performed instead of step S3 in the first embodiment. As shown in FIG. 10, the displacement extraction device 3a searches for a target image (step S11). Specifically, the image requesting unit 42 uses the target date and time information and area information related to the target image in the specification information to search for and acquire a sensor image corresponding to the target date and time information and area information from among the sensor images held by the image providing system 4, and outputs the acquired sensor image to the image checking unit 43. If the quality of the sensor image does not satisfy the conditions, the image checking unit 43 instructs the image requesting unit 42 to search for a sensor image again, and the image requesting unit 42 searches for a sensor image from the image providing system 4 again using the target date and time information and area information in the specification information. The image requesting unit 42 and the image checking unit 43 repeat these processes until a sensor image whose quality satisfies the conditions is obtained. The image request unit 42 may determine whether to search first for a SAR image or an optical image as an initial value, or may determine whether to search first for a SAR image or an optical image using the image type determination result by the image type determination unit 32 described in embodiment 1.

[0092] If there is a target image that satisfies the condition (Yes in step S12), the displacement extraction device 3a acquires the target image (step S13). That is, if there is a sensor image whose quality has been determined by the image confirmation unit 43 to satisfy the condition through the process of step S11, the displacement extraction device 3a outputs the sensor image to the target image acquisition unit 33, and the target image acquisition unit 33 acquires the target image.

[0093] Next, the displacement extraction device 3a searches for a reference image (step S14). In detail, the image requesting unit 42 searches for and acquires a sensor image corresponding to the target date and time information and area information from among the sensor images held by the image providing system 4, using the target date and time information and area information related to the reference image in the specification information, and outputs the acquired sensor image to the image checking unit 43. If the quality of the sensor image does not satisfy the conditions, the image checking unit 43 instructs the image requesting unit 42 to search for a sensor image again, and the image requesting unit 42 searches for a sensor image from the image providing system 4 again using the target date and time information and area information in the specification information. The image requesting unit 42 and the image checking unit 43 repeat these processes until a sensor image whose quality satisfies the conditions is obtained.

[0094] If there is a reference image that satisfies the condition (Yes in step S15), the displacement extraction device 3a acquires the reference image (step S16). That is, if there is a sensor image whose quality has been determined by the image confirmation unit 43 to satisfy the condition in the process of step S14, the sensor image is output to the target image acquisition unit 33, and the target image acquisition unit 33 acquires the reference image.

[0095] If the result of step S12 is No, the displacement extraction device 3a changes the type of image to be searched (step S17) and repeats the process from step S11. If the result of step S15 is No, the displacement extraction device 3a changes the type of image to be searched (step S18) and repeats the process from step S14.

[0096] As described above, the image requesting unit 42 acquires an image from the image providing system 4 based on the first target date and time information specifying the date and time the target image was captured and the area information, and outputs the acquired image to the target image acquiring unit 35 via the image confirming unit 43 as the target image. The image requesting unit 42 also acquires an image from the image providing system 4 based on the second target date and time information specifying the date and time the reference image was captured and the area information, and outputs the acquired image to the reference image acquiring unit 35 via the image confirming unit 43 as the reference image. If the image acquired by the image requesting unit 42 does not satisfy the specified quality conditions, the image confirming unit 43 instructs the image requesting unit 42 to re-acquire the image. The image requesting unit 42 may also acquire an image captured in the same season as the target image as the reference image.

[0097] Through the above processing, the displacement extraction device 3a acquires the target image and the reference image. The displacement extraction device 3a performs the processing shown in Fig. 10 instead of step S3 shown in Fig. 2, but other operations are the same as those in embodiment 1. That is, after acquiring the target image and the reference image, the displacement extraction device 3a performs the processing from step S4 shown in Fig. 2 onwards.

[0098] In this embodiment, the displacement extraction device 3a acquires a target image and a reference image that correspond to the conditions specified by the specification information, and extracts the displacement. The image type of the target image acquired by the displacement extraction device 3a through the above-described process is not necessarily the same as the image type of the reference image. However, the displacement extraction device 3a of this embodiment performs image conversion to extract the displacement when the image type of the target image and the image type of the reference image are not the same, as in the first embodiment. Therefore, as in the first embodiment, it is possible to increase the possibility of providing the user with a highly accurate displacement extraction result. Furthermore, in this embodiment, the user only needs to specify the conditions, and does not need to prepare sensor images themselves.

[0099] In addition to checking displacements before and after a specific event such as a disaster, a user may also want to check displacements for multiple dates and times, whether periodically or irregularly. For example, periodic monitoring may be desirable when monitoring similar locations other than the disaster site as high-risk locations following the occurrence of a specific disaster, or when checking long-term trends. For example, if a user wants to check the trend of displacements from 10 years ago, it would be time-consuming for the user to acquire sensor images corresponding to the 10-year period. In this embodiment, in such cases, the user can specify multiple target date and time information for the target images, allowing the displacement extraction device 3a to sequentially extract displacements. For example, the displacement extraction device 3a acquires an SAR image from 10 years ago as a reference image, and then acquires target images from 9 years ago, 8 years ago, and so on, one year at a time. However, there may be cases in which no SAR images capturing the target area exist within the 10-year period. In this embodiment, even in such cases, if optical images capturing the target area at the corresponding dates and times exist, the optical images can be converted into pseudo-SAR images and used to extract displacements.

[0100] Furthermore, because there are areas that are significantly affected by seasonal changes and areas that are less affected by seasonal changes, if the target area is an area that is significantly affected by seasonal changes, it is preferable that the target image and the reference image be captured in the same season. Therefore, in such cases, it is sufficient to specify the season when specifying the date and time of the reference image and select seasonal priority. Furthermore, if there is no reference image of the desired season, the image conversion unit 38 may perform seasonal change correction. The seasonal change correction may be performed using, for example, a machine learning method or any other method. In this case, the image conversion unit 38 may perform seasonal change correction even when the image type of the target image and the image type of the reference image are the same. In this example, the image conversion unit 38 functions as a seasonal change correction unit, but the displacement extraction device 3a may also include a seasonal change correction unit separate from the image conversion unit 38. That is, the displacement extraction device 3a may include a seasonal change correction unit that performs seasonal change correction on the reference image or target image when the season in which the target image was captured differs from the season in which the reference image was captured, and the displacement extraction unit 40 may extract the displacement using the reference image or target image after seasonal change correction.

[0101] <Variation 1> FIG. 11 is a diagram showing an example of the configuration of a displacement extraction system according to Modification 1 of this embodiment. A displacement extraction system 1b according to Modification 1 of this embodiment includes a displacement extraction device 3b and a user terminal 2. The user terminal 2 has the same configuration as the user terminal 2 according to Embodiment 1. The displacement extraction device 3b is similar to the displacement extraction device 3a shown in FIG. 8 except that it includes a specified information acquisition unit 31b instead of the specified information acquisition unit 31a. Components having the same functions as those in Embodiment 1 or the example shown in FIG. 8 are designated by the same reference numerals as those in Embodiment 1 or the example shown in FIG. 8, and redundant explanations will be omitted. Below, differences from Embodiment 1 or the example shown in FIG. 8 will be mainly explained.

[0102] As in the first embodiment, the user terminal 2 stores the target image in the image storage unit 22, and the transmitter / receiver 24 reads out the target image stored in the image storage unit 22 and transmits it to the displacement extraction device 3b. In the first modification of this embodiment, the user terminal 2 does not need to transmit the reference image to the displacement extraction device 3b.

[0103] In the first modification of this embodiment, the specification information includes target date and time information specifying the date and time of capturing the reference image, in addition to the region information and information indicating the displacement detection target described in the first embodiment. The specification information acquisition unit 31b receives the specification information from the user terminal 2 and outputs the received specification information to the image request unit 42 and the image type determination unit 32. In the first modification of this embodiment, the target image is provided by the user, and the displacement extraction device 3b acquires the reference image from the image providing system 4 based on the information specified by the specification information. Note that if the entire target image provided by the user is to be the displacement extraction target, the specification information does not need to include region information. In this case, the image request unit 42 acquires the reference image from the image providing system 4, regarding the region corresponding to the target image as the target region. The method of acquiring the reference image is the same as steps S14 to S18 of FIG. 10.

[0104] In a first variation of this embodiment, the image request unit 42 acquires an image from the image providing system 4 based on target date and time information specifying the date and time of capturing the reference image and information indicating the target area, and outputs the acquired image to the reference image acquisition unit 35 as a reference image. The information indicating the target area may be area information in the specification information or information indicating the area of ​​the target image added to the target image. In the first variation of this embodiment, the user only needs to provide the target image and does not need to prepare a reference image. When a specific event occurs, the user may acquire a target image after the event and check the target image independently. Therefore, although the user may store the target image in the user terminal 2 or have the right to use the target image, it is time-consuming for the user to prepare a reference image. In this embodiment, the user only needs to specify the conditions for the reference image, so the user can obtain displacement extraction results without any effort while utilizing target images that the user owns or has the right to use.

[0105] Furthermore, if the user has sensor images other than the target image in the user terminal 2, the user may also transmit these sensor images to the displacement extraction device 3b. In this case, the displacement extraction device 3b can also use sensor images acquired from the user terminal 2 as candidates for the reference image. Furthermore, if the image providing system 4 is a system managed by the operator of the displacement extraction device 3b, the target image acquired from the user terminal 2 and sensor images other than the target image can be stored in the image providing system 4, thereby increasing the number of sensor images stored in the image providing system 4. As described above, the method for the user to provide sensor images may also be a method using a recording medium.

[0106] The displacement extraction device 3a shown in FIG. 8 and the displacement extraction device 3b shown in FIG. 11 are realized, for example, by the computer system shown in FIG. 7, similar to the displacement extraction device 3 of the first embodiment. The image request unit 42 is realized by the control unit 101 and communication unit 105 shown in FIG. 7, and the image confirmation unit 43 is realized by the control unit 101 shown in FIG. 7. The user terminal 2a is also realized, for example, by the computer system shown in FIG. 7. The displacement extraction devices 3a and 3b may be realized by multiple computer systems. For example, the displacement extraction devices 3a and 3b may be realized by a cloud computer system.

[0107] Embodiment 3 FIG. 12 is a diagram showing an example of the configuration of a displacement extraction system according to the third embodiment. The displacement extraction system 1c of this embodiment includes a displacement extraction device 3c and a user terminal 2a. The user terminal 2a is similar to the user terminal 2a of the second embodiment. The displacement extraction device 3c adds an emergency observation request unit 44 to the displacement extraction device 3a of the second embodiment, and includes a designated information acquisition unit 31c instead of the designated information acquisition unit 31a. Components having the same functions as those of the second embodiment are given the same reference numerals as those of the second embodiment, and redundant explanations will be omitted. Below, differences from the second embodiment will be mainly explained.

[0108] In this embodiment, an example will be described in which a request for emergency observation by a satellite is made due to the occurrence of a large-scale disaster or the like. In satellite observation, emergency observation for observing a specific area may be accepted in addition to normal observation. For example, emergency observation is performed when a large-scale disaster occurs. The displacement extraction device 3c of this embodiment requests emergency observation by a satellite 7 in response to a user request.

[0109] In this embodiment, the specification information includes an observation request indicating a desire to acquire a target image by emergency observation. The observation request includes area information indicating the target area for which imaging is requested. The observation request may also include a desired date and time for imaging. The observation request may also include the type of sensor for which observation is requested.

[0110] The designated information acquisition unit 31c outputs the observation request included in the designated information to the emergency observation request unit 44. The emergency observation request unit 44 transmits the observation request, which requests that the target area be observed by a sensor mounted on the satellite, to the satellite observation operation system 6, which accepts requests for satellite observation. Specifically, the emergency observation request unit 44 transmits the emergency observation request received from the designated information acquisition unit 31c to the satellite observation operation system 6.

[0111] If the satellite observation operation system 6 is able to perform observation corresponding to the observation request received from the displacement extraction device 3c, it sends a command to the satellite 7, causing the sensor on board the satellite 7 to take an image corresponding to the observation request. Note that a ground station or the like that sends commands to the satellite 7 is typically provided separately from the satellite observation operation system 6, and the command is sent via the ground station, but the illustration of the ground station and other devices between the satellite observation operation system 6 and the satellite 7 is omitted. When the sensor on board the satellite 7 takes an image corresponding to the observation request, the image providing system 4 acquires the sensor image sent from the satellite 7. Note that a ground station that receives observation data corresponding to the sensor image, and a processing system that processes the observation data to create a sensor image to be provided are also provided between the satellite 7 and the image providing system 4, but these are also omitted from the illustration.

[0112] When a sensor image corresponding to an observation request is stored in the image providing system 4, the image requesting unit 42 acquires the sensor image as a target image. The image checking unit 43 outputs the target image acquired by the image requesting unit 42 to the target image acquiring unit 33. As a result, the target image acquiring unit 33 acquires the image acquired by the observation corresponding to the observation request.

[0113] The method of acquiring the reference image in this embodiment is the same as that in embodiment 2. Note that the displacement extraction device 3c may also acquire the reference image by sending an observation request in the same manner. Furthermore, the processing performed by the displacement extraction device 3c in this embodiment after acquiring the target image and the reference image is the same as that in embodiment 1.

[0114] The displacement extraction device 3c of this embodiment is realized, for example, by the computer system shown in Fig. 7, similar to the displacement extraction device 3 of the first embodiment. The displacement extraction device 3c may be realized by a plurality of computer systems. For example, the displacement extraction device 3c may be realized by a cloud computer system.

[0115] In this embodiment, when a user wishes to acquire a target image by emergency observation and obtain a displacement extraction result, the displacement extraction device 3c transmits an observation request in response to the user's request, allowing the user to obtain a displacement extraction result using the latest target image. Also, in this embodiment, since the target image may be either a SAR image or an optical image, by specifying multiple sensor types, such as SAR or optical sensor, when transmitting an observation request to the satellite observation operation system 6, the possibility of satellite observation being realized can be increased compared to when a single sensor is specified.

[0116] Embodiment 4 FIG. 13 is a diagram showing an example of the configuration of a displacement extraction system according to the fourth embodiment. A displacement extraction system 1d according to this embodiment includes a displacement extraction device 3d and a user terminal 2a. The user terminal 2a is similar to the user terminal 2a according to the second embodiment. The displacement extraction device 3d includes a ground observation request unit 45 in addition to the displacement extraction device 3a according to the second embodiment, and includes a specified information acquisition unit 31d instead of the specified information acquisition unit 31a. Components having the same functions as those according to the second embodiment are assigned the same reference numerals as those according to the second embodiment, and redundant explanations will be omitted. Below, differences from the second embodiment will be mainly explained.

[0117] In the event of a large-scale disaster, emergency observation may be performed using a satellite 7 as described in the third embodiment, but satellite observation generally has a low degree of flexibility. For this reason, in the event of a small-scale disaster or monitoring triggered by some event, it is expected that ground-based observation, which has greater operational flexibility, will be performed. In this embodiment, the displacement extraction device 3d requests ground-based observation in response to a user request.

[0118] In this embodiment, the specification information includes a ground observation request indicating that a target image is desired to be acquired by ground observation. The ground observation request includes area information indicating the target area for which imaging is requested. The ground observation request may also include a desired date and time for imaging. The ground observation request may also include the type of sensor for which observation is requested.

[0119] The designated information acquisition unit 31d outputs the ground observation request included in the designated information to the ground observation request unit 45. The ground observation request unit 45 transmits the ground observation request, which requests that the target area be observed by a sensor mounted on the aircraft, to the ground observation operation system 8, which accepts requests for aircraft observation. Specifically, the ground observation request unit 45 transmits the ground observation request received from the designated information acquisition unit 31d to the ground observation operation system 8.

[0120] If the ground observation operation system 8 is able to perform the observation corresponding to the ground observation request received from the displacement extraction device 3d, it causes an observation device 9, such as an aircraft equipped with a sensor, to capture an image of the target area requested in the ground observation request. The image providing system 4 acquires the sensor images acquired by the observation device 9. The sensor images acquired by the observation device 9 may be provided on a recording medium.

[0121] When a sensor image corresponding to a ground observation request is stored in the image providing system 4, the image requesting unit 42 acquires the sensor image as a target image. The image checking unit 43 outputs the target image acquired by the image requesting unit 42 to the target image acquiring unit 33. As a result, the target image acquiring unit 33 acquires the image acquired by the observation corresponding to the ground observation request as the target image.

[0122] The method of acquiring the reference image in this embodiment is the same as that in embodiment 2. Note that the displacement extraction device 3d may also acquire the reference image by transmitting a ground observation request in the same manner. Furthermore, the processing performed by the displacement extraction device 3d in this embodiment after acquiring the target image and the reference image is the same as that in embodiment 1.

[0123] Ground observations may be performed multiple times, either periodically or irregularly. For example, if a slope collapse occurs due to heavy rain, it may be necessary to observe the same area multiple times, either periodically or irregularly. In such cases, multiple observations may be requested by specifying a regular interval in the ground observation request sent from the user terminal 2a or by specifying multiple observation dates and times.

[0124] The displacement extraction device 3d of this embodiment is realized, for example, by the computer system shown in Fig. 7, similar to the displacement extraction device 3 of the first embodiment. The displacement extraction device 3d may be realized by a plurality of computer systems. For example, the displacement extraction device 3d may be realized by a cloud computer system.

[0125] In this embodiment, when a user wishes to acquire a target image through terrestrial observation and obtain a displacement extraction result, the displacement extraction device 3d transmits an observation request in response to the user's request, allowing the user to obtain a displacement extraction result using the latest target image. Furthermore, in this embodiment, since the target image may be an SAR image or an optical image, specifying multiple sensor types, such as SAR or optical sensor, when transmitting an observation request to the terrestrial observation operation system 8 increases the likelihood of satellite observation compared to specifying a single sensor. Generally, for terrestrial observation, observation using a drone is advantageous in terms of cost, and optical images are often acquired using a drone. On the other hand, as described in the first embodiment, SAR images can be acquired even at night or on cloudy days, so it is expected that SAR images will often be obtained as search results when searching for a reference image corresponding to a target area. In this embodiment, even if the image type of the target image and the image type of the optical image are different, image conversion is performed to extract the displacement, thereby increasing the likelihood of providing the user with highly accurate displacement extraction results.

[0126] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or different embodiments may be combined with each other. It is also possible to omit or modify parts of the configurations as long as they do not deviate from the gist of the invention. [Explanation of symbols]

[0127] 1, 1a, 1b, 1c, 1d Displacement extraction system, 2, 2a User terminal, 3, 3a, 3b, 3c, 3d Displacement extraction device, 4 Image provision system, 5 Learning device, 6 Satellite observation operation system, 7 Satellite, 8 Ground observation operation system, 9 Observation device, 21, 51 Image acquisition unit, 22 Image storage unit, 23 Input acceptance unit, 24 Transmitting / receiving unit, 25 Display unit, 31, 31a, 31b, 31c, 31d Designated information acquisition unit, 32 Image type determination unit, 33 Target image acquisition unit, 34 Target image extraction unit, 35 Reference image acquisition unit, 36 Reference image extraction unit, 37 Conversion necessity determination unit, 38 Image conversion unit, 39 Alignment unit, 40 Displacement extraction unit, 41 Output unit, 42 Image request unit, 43 Image confirmation unit, 44 Emergency observation request unit, 45 Ground observation request unit, 52 learning data generation unit, 53 model generation unit, 54, 382 learned model memory unit, 381 inference unit, 501 input layer, 502 intermediate layer, 503 output layer, 531 pixel value comparison unit, 532 parameter adjustment unit, 533 memory.

Claims

1. a target image acquisition unit that acquires a target image that is an image of a target area that is a target for displacement extraction, the image type of the image of the target area being at least one of a sensor image of an SAR image acquired by a synthetic aperture radar and an optical image; a reference image acquisition unit that acquires a reference image whose image type of the image of the target area is at least one of a SAR image and an optical image acquired by a synthetic aperture radar, and whose image acquisition date and time are different from those of the target image; an image conversion unit; a displacement extraction unit that extracts the displacement; an output unit that outputs an extraction result by the displacement extraction unit; Equipped with When the target image acquisition unit can acquire the target image that is an optical image and the target image that is an SAR image, the target image acquisition unit acquires the target image of a priority type that is an image type corresponding to the type of a target for which displacement is to be detected, from the target images that can be acquired, When the reference image acquisition unit can acquire the reference image that is an optical image and the reference image that is an SAR image, the reference image acquisition unit acquires the reference image of the priority type from among the acquireable reference images, when an image type of the target image acquired by the target image acquisition unit is different from an image type of the reference image acquired by the reference image acquisition unit, the image conversion unit performs image conversion to convert an image of the target image acquired by the target image acquisition unit and the reference image acquired by the reference image acquisition unit, the image type of which is different from the priority type, into an image of the priority type, thereby generating a converted image; The displacement extraction unit, when the image type of the target image acquired by the target image acquisition unit is different from the image type of the reference image acquired by the reference image acquisition unit, extracts the displacement by comparing the converted image with the target image acquired by the target image acquisition unit and the reference image acquired by the reference image acquisition unit, which have the image type of the priority type; and when the image type of the target image acquired by the target image acquisition unit is the same as the image type of the reference image acquired by the reference image acquisition unit, extracts the displacement by comparing the target image acquired by the target image acquisition unit with the reference image acquired by the reference image acquisition unit.

2. an image type determination unit that determines the priority type; The displacement extraction device according to claim 1 , further comprising:

3. 3. The displacement extraction device according to claim 2, wherein the image type determination unit determines the priority type based on an input from a user.

4. 3. The displacement extraction device according to claim 2, wherein the image type determination unit determines the priority type depending on the type of object of displacement detection.

5. a target image acquisition unit that acquires a target image, which is an image of a target region from which displacement is extracted; a reference image acquisition unit that acquires a reference image that is an image of the target area and has a different image capture date and time than the target image; an image type determination unit that determines a priority type of image that is not subjected to image conversion when the image types of the target image and the reference image are different, in accordance with the type of displacement detection target; an image conversion unit that generates a converted image by converting an image of the reference image and the target image whose image type is different from the priority type into an image of the priority type when the image type of the reference image and the target image is different; a displacement extraction unit that extracts a displacement by comparing the converted image with an image of the priority type among the reference image and the target image when the image type of the target image is different from the image type of the reference image, and extracts a displacement by comparing the target image acquired by the target image acquisition unit with the reference image acquired by the reference image acquisition unit when the image type of the target image is the same as the image type of the reference image; an output unit that outputs an extraction result by the displacement extraction unit; Equipped with the priority type is a SAR image acquired by a synthetic aperture radar or an optical image acquired by an optical sensor; the target image is a SAR image or an optical image; The displacement extraction device is characterized in that the reference image is a SAR image or an optical image.

6. 6. The displacement extraction device according to claim 1, wherein the target image acquisition unit acquires the target image by receiving the target image from a user terminal.

7. The displacement extraction device according to claim 6 , wherein the reference image acquisition unit acquires the reference image by receiving the reference image from a user terminal.

8. an image requesting unit that acquires an image from an image providing system based on target date and time information, which is information regarding the date and time of capturing the reference image, and information indicating the target area, and outputs the acquired image to the reference image acquiring unit as the reference image; The displacement extraction device according to claim 6, further comprising:

9. The displacement extraction device according to claim 8 , wherein the target date and time information includes an allowable range for the date and time.

10. an image requesting unit that acquires an image from an image providing system based on first target date and time information that specifies the date and time when the target image was captured and information that indicates the target area, and outputs the acquired image to the target image acquiring unit as the target image, and acquires an image from the image providing system based on second target date and time information that specifies the date and time when the reference image was captured and information that indicates the target area, and outputs the acquired image to the reference image acquiring unit as the reference image; 6. The displacement extraction device according to claim 1, further comprising:

11. an image confirmation unit that instructs the image request unit to re-acquire an image if the image acquired by the image request unit does not satisfy a predetermined condition regarding quality; 11. The displacement extraction device according to claim 8, further comprising:

12. 12. The displacement extraction device according to claim 8, wherein the image requesting unit acquires, as the reference image, an image captured in the same season as the target image.

13. a seasonal change correction unit that corrects seasonal changes in the reference image or the target image when the season in which the target image was captured differs from the season in which the reference image was captured; 13. The displacement extraction device according to claim 1, wherein the displacement extraction unit extracts the displacement using the reference image or the target image after seasonal changes have been corrected.

14. an emergency observation request unit that transmits an observation request to a satellite observation operation system that receives requests for observation by the satellite, the observation request requesting that the target area be observed by a sensor mounted on the satellite; Equipped with The displacement extraction device according to claim 1 , wherein the target image acquisition unit acquires, as the target image, an image acquired by observation corresponding to the observation request.

15. a ground observation request unit that transmits a ground observation request requesting that the target area be observed by a sensor mounted on an aircraft to a ground observation operation system that accepts requests for observation by the aircraft; Equipped with The displacement extraction device according to claim 1 , wherein the target image acquisition unit acquires, as the target image, an image acquired by observation corresponding to the ground observation request.

16. The displacement extraction device according to claim 15, wherein the aircraft is an unmanned aerial vehicle.

17. The priority type is a SAR image, The displacement extraction device described in claim 16, characterized in that the image conversion unit converts the target image into an SAR image when the target image acquired by the target image acquisition unit is an optical image and the reference image acquired by the reference image acquisition unit is an SAR image.

18. a target image acquisition unit that acquires a target image, which is an image of a target region from which displacement is extracted; a reference image acquisition unit that acquires a reference image that is an image of the target area and has a different image capture date and time than the target image; an image conversion unit that generates a converted image by converting a first image, which is one of the reference image and the target image, into an image of an image type of a second image, which is the other image, when the image type of the reference image and the target image differs; a displacement extraction unit; an output unit that outputs an extraction result by the displacement extraction unit; Equipped with the image type is a SAR image acquired by a synthetic aperture radar or an optical image acquired by an optical sensor, the image transformation unit acquires a probability map indicating a distribution of the probability of the image transformation; The displacement extraction device is characterized in that, when the image type of the target image and the image type of the reference image are different, the displacement extraction unit extracts the displacement using the comparison result between the second image and the converted image and the likelihood map, and when the image type of the target image and the image type of the reference image are the same, the displacement extraction unit extracts the displacement by comparing the target image acquired by the target image acquisition unit with the reference image acquired by the reference image acquisition unit.

19. a target image acquisition unit that acquires a target image, which is an image of a target region from which displacement is extracted; a reference image acquisition unit that acquires a reference image that is an image of the target area and has a different image capture date and time than the target image; an image conversion unit that generates a converted image by converting a first image, which is one of the reference image and the target image, into an image of an image type of a second image, which is the other image, when the image type of the reference image and the target image differs; a displacement extraction unit that extracts the displacement by comparing the second image with the converted image when the image type of the target image is different from the image type of the reference image, and extracts the displacement by comparing the target image acquired by the target image acquisition unit with the reference image acquired by the reference image acquisition unit when the image type of the target image is the same as the image type of the reference image; an output unit that outputs an extraction result by the displacement extraction unit; Equipped with the image type is a SAR image acquired by a synthetic aperture radar or an optical image acquired by an optical sensor, the image transformation unit acquires a probability map indicating a distribution of the probability of the image transformation; The displacement extraction device is characterized in that the output unit outputs the extraction result and the likelihood map in association with each other.

20. A user terminal; a displacement extraction device; Equipped with The displacement extraction device a target image acquisition unit that acquires a target image, which is an image of a target region from which displacement is extracted; a reference image acquisition unit that acquires a reference image that is an image of the target area and has a different image capture date and time than the target image; an image type determination unit that determines a priority type of image that is not subjected to image conversion when the image types of the target image and the reference image are different, in accordance with the type of displacement detection target; an image conversion unit that generates a converted image by converting an image of the reference image and the target image whose image type is different from the priority type into an image of the priority type when the image type of the reference image and the target image is different; a displacement extraction unit that extracts a displacement by comparing the converted image with an image of the priority type among the reference image and the target image when the image type of the target image is different from the image type of the reference image, and extracts a displacement by comparing the target image acquired by the target image acquisition unit with the reference image acquired by the reference image acquisition unit when the image type of the target image is the same as the image type of the reference image; an output unit that transmits the extraction result by the displacement extraction unit to the user terminal; Equipped with the user terminal displays the extraction result received from the displacement extraction device; the priority type is a SAR image acquired by a synthetic aperture radar or an optical image acquired by an optical sensor; the target image is a SAR image or an optical image; The displacement extraction system is characterized in that the reference image is a SAR image or an optical image.

21. A displacement extraction method for a displacement extraction device, comprising: obtaining a target image, which is an image of a target region from which displacements are to be extracted; acquiring a reference image that is an image of the target area and has a different image capture date and time than the target image; determining a priority image type, which is an image type for which image conversion is not performed when the image types of the target image and the reference image are different, according to the type of object for which displacement is to be detected; generating a converted image by converting an image of the reference image and the target image, which has an image type different from the priority type, from among the reference image and the target image, to an image of the priority type when the image type of the reference image and the target image is different from the priority type; extracting a displacement by comparing the converted image with an image of the priority type among the reference image and the target image when the image type of the target image is different from the image type of the reference image, and extracting a displacement by comparing the acquired target image with the acquired reference image when the image type of the target image is the same as the image type of the reference image; outputting the displacement extraction result; Including, the priority type is a SAR image acquired by a synthetic aperture radar or an optical image acquired by an optical sensor; the target image is a SAR image or an optical image; The displacement extraction method is characterized in that the reference image is a SAR image or an optical image.

22. In the computer system, obtaining a target image, which is an image of a target region from which displacements are to be extracted; acquiring a reference image that is an image of the target area and has a different image capture date and time than the target image; determining a priority image type, which is an image type for which image conversion is not performed when the image types of the target image and the reference image are different, according to the type of object for which displacement is to be detected; generating a converted image by converting an image of the reference image and the target image, which has an image type different from the priority type, from among the reference image and the target image, to an image of the priority type when the image type of the reference image and the target image is different from the priority type; extracting a displacement by comparing the converted image with an image of the priority type among the reference image and the target image when the image type of the target image is different from the image type of the reference image, and extracting a displacement by comparing the acquired target image with the acquired reference image when the image type of the target image is the same as the image type of the reference image; outputting the displacement extraction result; Execute the priority type is a SAR image acquired by a synthetic aperture radar or an optical image acquired by an optical sensor; the target image is a SAR image or an optical image; The computer program product is characterized in that the reference image is a SAR image or an optical image.

Images