Alignment system, alignment method, and alignment program

Abstract

An alignment system (1) comprises an input image generation device (10) and a learning device (20). The input image generation device (10) comprises a simulated image generation unit that generates a first simulated image indicating a target range and corresponding to a first domain on the basis of first ground surface information indicating a ground surface in the target range, and generates a second simulated image indicating the target range and corresponding to a second domain on the basis of second ground surface information indicating the ground surface in the target range. The learning device (20) comprises an image deformation unit that deforms at least one of the first simulated image and the second simulated image on the basis of the amount of positional deviation set between the first simulated image and the second simulated image.

Description

Alignment System, Alignment Method, and Alignment Program

[0002] ,

[0006]

[0001] The present disclosure relates to an alignment system, an alignment method, and an alignment program.

[0002] When utilizing a plurality of satellite images with corresponding domains being different from each other, alignment needs to be performed. Patent Document 1 discloses a technique for performing alignment between a SAR image and an optical image using an inference model.

[0003] Japanese Patent No. 7262679

[0004] In the technique disclosed in Patent Document 1, actually acquired remote sensing images are utilized. However, in the field of disaster prevention, optical images are stored in archives, and a scenario where remote sensing images corresponding to a desired domain are not actually acquired, such as SAR (Synthetic Aperture Radar) images being acquired immediately after a disaster occurs while optical images are stored in archives, is realistic. The present disclosure aims to construct an inference model that can be utilized for performing alignment in the case where remote sensing images are not actually acquired, regarding a technique for performing alignment between remote sensing images with corresponding domains being different from each other.

[0005] The alignment system according to the present disclosure includes an input image generation device including a simulation image generation unit that, based on first ground surface information indicating the ground surface in a target range, generates, as a first simulation image, a remote sensing image that indicates the target range and corresponds to a first domain, and based on second ground surface information indicating the ground surface in the target range, generates, as a second simulation image, a remote sensing image that indicates the target range and corresponds to a second domain different from the first domain; and a learning device including an image deformation unit that deforms at least one of the first simulation image and the second simulation image based on a displacement amount when a displacement amount indicating the amount of displacement set between the first simulation image and the second simulation image is generated.

[0006] According to this disclosure, the simulated image generation unit of the input image generation device generates two types of simulated images corresponding to two types of domains. The image deformation unit of the learning device deforms at least one of the two types of simulated images based on the generated positional displacement amount. Here, an inference model can be constructed using the training data consisting of the two deformed types of simulated images and the positional displacement amount. Therefore, by utilizing this disclosure, it is possible to construct an inference model that can be used to perform alignment when remote sensing images have not actually been acquired, in relation to a technology for performing alignment between remote sensing images where the corresponding domains are different.

[0007] A diagram showing an example configuration of the alignment system 1 according to Embodiment 1. A diagram showing an example configuration of the input image generation device 10 according to Embodiment 1. A diagram showing an example configuration of the learning device 20 according to Embodiment 1. A diagram showing an example configuration of the inference device 30 according to Embodiment 1. A diagram showing an example hardware configuration of each device according to Embodiment 1. A flowchart showing the operation of the alignment system 1 in the learning phase according to Embodiment 1. A flowchart showing the operation of the alignment system 1 in the inference phase according to Embodiment 1. A diagram showing an example hardware configuration of each device according to a modified version of Embodiment 1. A diagram showing an example configuration of the input image generation device 10 according to Embodiment 2.

[0008] In the description and drawings of the embodiments, the same elements and corresponding elements are denoted by the same reference numeral. The descriptions of elements denoted by the same reference numeral are omitted or simplified as appropriate. The arrows in the figures mainly indicate the flow of data or processing. Also, "part" may be read as "circuit," "device," "equipment," "process," "step," "procedure," "processing," or "circuitry" as appropriate. The functions of each part of each device may be realized by firmware, software, hardware, or a combination thereof.

[0009] Embodiment 1. This embodiment will be described in detail below with reference to the drawings.

[0010] ***Configuration Description*** Figure 1 shows an example of the configuration of the alignment system 1 according to this embodiment. As shown in Figure 1, the alignment system 1 comprises an input image generation device 10, a learning device 20, an inference device 30, and a trained model storage unit 204. The multiple devices comprising the alignment system 1 may be configured integrally as appropriate. In the following, as an example of a combination of remote sensing images whose corresponding domains are different from each other, a combination of optical satellite images and radar satellite images will be described. However, this embodiment may also be applied to combinations of other types of remote sensing images. Specific examples of remote sensing images include optical images, SAR (synthetic aperture radar) images, or infrared images. Remote sensing images may be images taken from an aircraft or images taken from an artificial satellite.

[0011] Figure 2 shows an example configuration of the input image generation device 10. The input image generation device 10 includes a simulated image generation unit 101, as shown in Figure 2. Based on first ground surface information, the simulated image generation unit 101 simulates generating a first simulated image, which shows the target area and corresponds to a first domain. Based on second ground surface information, the simulated image generation unit 101 simulates generating a second simulated image, which shows the target area and corresponds to a second domain. The simulated image generation unit 101 uses simulation technology when generating the simulated images. The target area is any area of ​​the ground surface. The target area can be set in any way and may be changed as appropriate. The first ground surface information and the second ground surface information are information that shows the ground surface in the target area. At least one of the first ground surface information and the second ground surface information may be model information that shows a model of the ground surface in the target area. "Simulated image" is a general term for various simulated remote sensing images. Information regarding remote sensing images may include information about the sensor that captures the remote sensing image, and may also include information about the aircraft or satellite on which the sensor is mounted. The simulated image generation unit 101 may generate a first simulated image based on information indicating a first imaging condition if the first ground surface information is model information. The simulated image generation unit 101 may generate a second simulated image based on information indicating a second imaging condition if the second ground surface information is model information. The first imaging condition is the imaging condition for a remote sensing image corresponding to a first domain. The second imaging condition is the imaging condition for a remote sensing image corresponding to a second domain. The first domain is, in specific examples, the domain of an optical image. The second domain is a domain different from the first domain. The second domain is, in specific examples, the domain of a synthetic aperture radar image. As a specific example, the simulated image generation unit 101 takes model information DIN1, optical satellite information DIN2, and radar satellite information DIN3 as inputs, generates an optical satellite simulated image D101A based on model information DIN1 and optical satellite information DIN2, and generates a radar satellite simulated image D101B based on model information DIN1 and radar satellite information DIN3.Subsequently, the simulated image generation unit 101 outputs an optical satellite simulated image D101A and a radar satellite simulated image D101B. Specifically, the simulated image generation unit 101 generates each simulated image using at least one of the following methods: a method using Image Translation AI (Artificial Intelligence), a legacy method, and a method using an RCS (Radar Cross-Section) simulator from a model of the Earth's surface. Specific examples of the method using Image Translation AI include methods using refinement, GAN (Generative Adversarial Network), or GSG (Global Scene-harmonious Guidance). The legacy method, as a concrete example, is a method for simulating an optical image by artificially adjusting at least one of the luminance value and SN (Signal-Noise) value of a given optical image. The legacy method, as a concrete example, consists of at least one of the following: a method for converting an optical image to a grayscale image and adding noise to the grayscale image; a method for adjusting the contrast of an optical image; and a method for inverting the colors of an optical image. In the legacy method, the user may visually confirm and indicate the parameter values ​​related to the legacy method, and the simulated image generation unit 101 may set each parameter value according to the user's instructions. In other words, the simulated image generation unit 101 may generate a first simulated image from a target optical image by adjusting each parameter of the target optical image, which is an optical image indicating the target range. The simulated image generation unit 101 may also process the simulated image generated using the legacy method with AI and output the processed simulated image. The method using the RCS simulator is a method for simulating a SAR image using the RCS simulator based on the model information DIN1.

[0012] Model information DIN1 is information that indicates a model of the shape of the ground surface. Specific examples of model information DIN1 include DEM (Digital Elevation Model), DSM (Digital Surface Model), or a 3D model (such as a blender). The simulated image generation unit 101 may use only the model corresponding to the target area from among the models indicated by model information DIN1.

[0013] Optical satellite information DIN2 is information indicating the imaging conditions for the simulated optical satellite image D101A. Specifically, these imaging conditions consist of imaging parameters related to the optical sensor mounted on the target optical satellite, the first imaging time, and information indicating various external factors of the optical satellite at the first imaging time. Specifically, the imaging parameters indicated by optical satellite information DIN2 consist of parameters related to the band (spectroscopy), resolution, observation mode, brightness, observation angle, pointing angle (also called off-nadia angle), and imaging time. In the case of TDI (Time Delay Integration) imaging, the imaging time is the total integrated time (product of TDI number of stages and imaging period) which corresponds to the exposure time in a typical camera, and the TDI number of stages is set in the imaging parameters. The first imaging time is a virtual imaging time set as the imaging time for the first simulated image. External factors for the optical satellite at the time of the first imaging include, specifically, the season at the time of the first imaging, the solar altitude at the time of the first imaging, and the meteorological and atmospheric conditions around the target area at the time of the first imaging.

[0014] Radar satellite information DIN3 is information indicating the imaging conditions for the simulated radar satellite image D101B. These imaging conditions consist, specifically, information indicating imaging parameters related to the radar mounted on the target radar satellite, the second imaging time, and information indicating various external factors at the second imaging time. The radar is, in specific examples, a SAR. The imaging parameters indicated in radar satellite information DIN3 consist, specifically, parameters related to resolution, observation mode, observation angle, incidence angle, imaging azimuth angle, and radio wave irradiation time. Note that even for satellites with the same specifications, the resolution can vary depending on the pointing angle and incidence angle. Here, the name of the satellite taking the image and the imaging mode also correspond to imaging parameters, as the specifications are determined according to the satellite's name. For example, a satellite named "ALOS-2" uses L-band radio waves, and the approximate range of resolution is determined according to the imaging mode used, with further details determined by information such as the incidence angle. The second imaging time point is a hypothetical imaging time point set as the time point for capturing the second simulated image. Note that the first and second imaging time points may differ from each other. External factors at the second imaging time point include, as a specific example, the orbit of the radar satellite at the second imaging time point.

[0015] Optical satellite simulated image D101A corresponds to the first simulated image. Optical satellite simulated image D101A is a simulated optical image and also corresponds to a remote sensing image captured at the time of the first imaging.

[0016] Radar satellite simulated image D101B corresponds to the second simulated image. Radar satellite simulated image D101B is a simulated radar image and also corresponds to a remote sensing image taken at the second imaging time.

[0017] Figure 3 shows an example of the configuration of the learning device 20. As shown in Figure 3, the learning device 20 comprises a positional displacement amount generation unit 201, an image deformation unit 202, and a model generation unit 203.

[0018] The positional displacement amount generation unit 201 generates a positional displacement amount between two types of remote sensing images. Alternatively, the input image generation device 10 may also include the positional displacement amount generation unit 201 instead of the learning device 20. As a specific example, the positional displacement amount generation unit 201 generates a positional displacement amount to generate the positional displacement between an optical satellite simulated image D101A and a radar satellite simulated image D101B.

[0019] The positional displacement amount indicates the amount of positional displacement set between the first simulated image and the second simulated image, and also corresponds to the amount of displacement between the first simulated image and the second simulated image. The positional displacement amount may be a scalar value. The positional displacement amount may be a random value, a value that represents the positional displacement caused by the insufficient accuracy of the small satellite, or a value that represents the pixel displacement caused according to the features of the ground surface. Specific examples of ground surface features include the presence of artificial objects such as buildings, or the presence of topographic irregularities (such as mountains or valleys). In other words, the positional displacement amount generation unit 201 may calculate the positional displacement amount based on a DEM or DSM, or based on a 3D model, taking into account the collapse caused by elevation. As a specific example, the positional displacement amount generation unit 201 may set the positional displacement amount for mountainous areas and the positional displacement amount for flat areas to be different values.

[0020] The image deformation unit 202 simulates a pair of remote sensing images with known positional displacements by deforming at least one of the first simulated image and the second simulated image based on the positional displacement amount generated by the positional displacement amount generation unit 201. Specifically, the image deformation unit 202 takes at least one of the optical satellite simulated image D101A and the radar satellite simulated image D101B as input and deforms the input simulated image based on the positional displacement amount generated by the positional displacement amount generation unit 201.

[0021] The image deformation unit 202 may further deform at least one of the first and second simulated images so that, assuming that the ground surface changes in the target area between two time periods, one of the first and second simulated images becomes a remote sensing image showing the ground surface before the change, and the other of the first and second simulated images becomes a remote sensing image showing the ground surface after the change. In other words, the image deformation unit 202 may further deform at least one of the simulated images with the aim of generating a pair of remote sensing images corresponding to the ground surface before and after a change in the ground surface occurring in the target area between two time periods, as a pair of images for two time periods. Specific examples of changes in the ground surface include at least one of the following: changes in farmland or green spaces due to seasonal differences, changes in artificial objects due to the construction or demolition of buildings, changes in the water surface due to flooding, changes in relief caused by volcanic ejecta or landslides, and changes in topography caused by earthquakes. A pair of images for two time periods is a pair of a remote sensing image taken at one time and a remote sensing image taken at a time different from that time. As a specific example, the period in question is a period before the disaster, and the period different from the period in question is a period after the disaster. As another specific example, the period in question is a certain season, and the period different from the period in question is a different season. Furthermore, the image deformation unit 202 may add noise caused by the disturbance to at least one of the simulated images for the purpose of learning the effects of disturbances. In other words, the image deformation unit 202 may further reflect the effects of disturbances in at least one of the first and second simulated images by deforming at least one of the first and second simulated images, assuming that disturbances occur when remote sensing images are captured. Note that the disturbances corresponding to each simulated image do not have to be constant.

[0022] The model generation unit 203 generates a trained model 210 by learning the relationship between the first training image, the second training image, and the positional displacement amount using training data consisting of a first training image, a second training image, and the positional displacement amount. Here, if the first simulated image is deformed, the first training image is the deformed first simulated image. If the first simulated image is not deformed, the first training image is the first simulated image. If the second simulated image is deformed, the second training image is the deformed second simulated image. If the second simulated image is not deformed, the second training image is the second simulated image. As a specific example, the model generation unit 203 generates a trained model 210 by learning the relationship between the optical satellite simulated image D101A and the radar satellite simulated image D101B and the positional displacement amount occurring between the optical satellite simulated image D101A and the radar satellite simulated image D101B. This positional displacement amount corresponds to the correct answer. In this case, the model generation unit 203 may utilize machine learning or AI (Artificial Intelligence) technology, and may generate a trained model 210 in the same manner as the inference model in Embodiment 4 of Patent Document 1. The model generation unit 203 may generate the trained model 210 by semi-supervised learning or reinforcement learning. The model generation unit 203 stores the generated trained model 210 in the trained model storage unit 204.

[0023] The trained model storage unit 204 stores the trained model 210.

[0024] The trained model 210 is a model that infers the amount of positional displacement between two remote sensing images whose corresponding domains are different. In other words, the trained model 210 is a model that infers the amount of positional displacement corresponding to the first inference image and the second inference image from a first inference image which is a remote sensing image whose corresponding domain is the first domain and a second inference image which is a remote sensing image whose corresponding domain is the second domain. As a specific example, the trained model 210 is a model that infers the amount of positional displacement between a remote sensing image whose corresponding domain is an optical image and a remote sensing image whose corresponding domain is a SAR image. The amount of positional displacement inferred by the trained model 210 is used when performing alignment between the two remote sensing images. The trained model 210 may be an inference model specialized for imaging conditions or it may be a highly general-purpose inference model. An inference model specialized for imaging conditions is a model generated by using training data corresponding to the same or similar imaging conditions. This inference model can infer the amount of positional displacement between two remote sensing images corresponding to the same or similar imaging conditions as the training data for the inference model with relatively high accuracy. A highly versatile inference model is one that is generated by using training data corresponding to various imaging conditions. This inference model may be one that has further learned the relationship between each parameter corresponding to the imaging conditions and the amount of positional displacement, or it may be a model that uses each parameter corresponding to the imaging conditions as an auxiliary tool to infer the amount of positional displacement.

[0025] Figure 4 shows an example configuration of the inference device 30. The inference device 30 includes a positioning unit 301. The positioning unit 301 inputs two remote sensing images corresponding to different domains into a trained model 210 and infers the amount of positional misalignment between the two remote sensing images. Subsequently, the positioning unit 301 performs positional alignment between the two remote sensing images based on the inferred amount of positional misalignment. The two remote sensing images may be OpenEarthMap images, or images taken at different times, such as before and after a disaster. As a specific example, the positioning unit 301 takes an optical image DIN4 and a radar image DIN5 as inputs, inputs the optical image DIN4 and radar image DIN5 into the trained model 210, and obtains the amount of positional misalignment corresponding to the optical image DIN4 and radar image DIN5 from the trained model 210. Subsequently, the alignment unit 301 performs alignment between the optical image DIN4 and the radar image DIN5 based on the acquired positional misalignment amount, and outputs the processed optical image DOUT1 and the processed radar image DOUT2. Land cover classification may also be performed on each of the aligned remote sensing images.

[0026] Optical image DIN4 is a remote sensing image captured by the target optical satellite.

[0027] Radar image DIN5 is a remote sensing image captured by the target radar satellite.

[0028] The processed optical image DOUT1 is an image corresponding to the optical image DIN4, and represents the result of alignment between the optical image DIN4 and the radar image DIN5. The processed optical image DOUT1 may be the same as the optical image DIN4, or it may be an image obtained by processing the optical image DIN4.

[0029] The processed radar image DOUT2 is the image corresponding to the radar image DIN5, and represents the result of alignment between the optical image DIN4 and the radar image DIN5. The processed radar image DOUT2 may be the same as the radar image DIN5, or it may be an image obtained by processing the radar image DIN5.

[0030] Figure 5 shows an example of the hardware configuration of each device according to this embodiment. Each device consists of a computer. Each device may consist of multiple computers.

[0031] As shown in this figure, each device is a computer equipped with hardware such as a processor 51, memory 52, auxiliary storage device 53, input / output interface (IF) 54, and communication device 55. These hardware components are connected as appropriate via signal lines 59.

[0032] The processor 51 is an IC (Integrated Circuit) that performs arithmetic processing and controls the hardware of the computer. Specific examples of the processor 51 include a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or a GPU (Graphics Processing Unit). Each device may have multiple processors that substitute for the processor 51. The multiple processors share the role of the processor 51.

[0033] Memory 52 is typically a volatile storage device, specifically RAM (Random Access Memory). Memory 52 is also called main memory. Data stored in memory 52 is saved to auxiliary storage device 53 as needed.

[0034] The auxiliary storage device 53 is typically a non-volatile storage device, specifically a ROM (Read Only Memory), HDD (Hard Disk Drive), or flash memory. Data stored in the auxiliary storage device 53 is loaded into the memory 52 as needed. The memory 52 and the auxiliary storage device 53 may be configured as an integrated unit.

[0035] Input / Output IF54 is a port to which input and output devices are connected. A specific example of an input / output IF54 is a USB (Universal Serial Bus) terminal. Specific examples of input devices include a keyboard and mouse. Specific examples of output devices include a display.

[0036] The communication device 55 is a receiver and a transmitter. Specific examples of the communication device 55 include a communication chip or a NIC (Network Interface Card).

[0037] Each part of each device may use the input / output IF 54 and the communication device 55 as appropriate when communicating with other devices.

[0038] The auxiliary storage device 53 stores the alignment program. The alignment program is a program that enables the computer to implement the functions of each part of each device. The alignment program is loaded into memory 52 and executed by the processor 51.

[0039] Data used when executing the alignment program, and data obtained by executing the alignment program, are appropriately stored in the memory device. Each part of each device utilizes the memory device as appropriate. The memory device consists of, as a specific example, memory 52, auxiliary storage device 53, registers in the processor 51, and at least one of the cache memory in the processor 51. Note that the terms data and information may have the same meaning. The memory device may be independent of the computer. The functions of memory 52 and auxiliary storage device 53 may be implemented by other memory devices.

[0040] The alignment program may be recorded on a computer-readable non-volatile recording medium. Specific examples of non-volatile recording media include optical discs or flash memory. The alignment program may also be provided as a program product.

[0041] ***Description of Operations*** The operation procedures of each device constituting the alignment system 1 are collectively referred to as an alignment method. Also, the programs for realizing the operations of each device constituting the alignment system 1 are collectively referred to as an alignment program.

[0042] Fig. 6 is a flowchart showing an example of the operation of the alignment system 1 in the learning phase. The operation will be described using Fig. 6.

[0043] (Step S101) The simulated image generation unit 101 takes the model information DIN1, the optical satellite information DIN2, and the radar satellite information DIN3 as inputs, and generates an optical satellite simulated image D101A and a radar satellite simulated image D101B.

[0044] (Step S102) The misalignment amount generation unit 201 generates a misalignment amount corresponding to the misalignment set between the optical satellite simulated image D101A and the radar satellite simulated image D101B.

[0045] (Step S103) The image deformation unit 202 appropriately deform at least one of the optical satellite simulated image D101A and the radar satellite simulated image D101B based on the misalignment amount generated by the misalignment amount generation unit 201. The image deformation unit 202 may further deform at least one of the optical satellite simulated image D101A and the radar satellite simulated image D101B assuming at least one of the changes in the ground surface occurring between two times and disturbances.

[0046] (Step S104) The model generation unit 203 generates a learned model 210 using the learning data composed of the optical satellite simulated image D101A, the radar satellite simulated image D101B, and the misalignment amount corresponding to the optical satellite simulated image D101A and the radar satellite simulated image D101B. The model generation unit 203 stores the generated learned model 210 in the learned model storage unit 204.

[0047] Fig. 7 is a flowchart showing an example of the operation of the alignment system 1 in the inference phase. The operation will be described using Fig. 7.

[0048] (Step S111) The alignment unit 301 acquires the learned model 210 from the learned model storage unit 204, and inputs the optical image DIN4 and the radar image DIN5 into the learned model 210, thereby inferring the amount of misalignment between the optical image DIN4 and the radar image DIN5. Thereafter, the alignment unit 301 performs alignment between the optical image DIN4 and the radar image DIN5 based on the inferred amount of misalignment, and outputs the processed optical image DOUT1 and the processed radar image DOUT2.

[0049] ***Explanation of the Effects of Embodiment 1*** In the prior art, when a sufficient amount of remote sensing images has not been actually acquired, alignment cannot be performed on different types of remote sensing images, and as a result, analysis results cannot be compared between different types of remote sensing images. On the other hand, in the present embodiment, a pair of different types of remote sensing images is simulatedly generated as a pair of simulated images, and an inference model for inferring the amount of misalignment between different types of remote sensing images is generated based on the generated pair of simulated images. Therefore, according to the present embodiment, when a pair of different types of remote sensing images, such as a pair of an optical image and a SAR image, is given, by utilizing the inference model, a remote sensing image can be generated such that the amount of misalignment between each pixel in the image becomes relatively small.

[0050] Also, in the present embodiment, instead of preparing a pair of images at two times by actually imaging a remote sensing image, simulated images before and after deformation can be used as a pair of images at two times. Therefore, according to the present embodiment, there is an advantage that the amount of data set that can be secured increases. Further, according to the present embodiment, for the purpose of performing an inference considering changes in the ground surface or disturbances between two times, a simulated image different from the simulated image after change and the simulated image after change can be appropriately generated as learning data to generate an inference model. By utilizing the inference model, it becomes possible to estimate the amount of misalignment with relatively high accuracy.

[0051] ***Other Configurations*** <Modification 1> Figure 8 shows an example of the hardware configuration of each device according to this modification. Each device is equipped with a processor 51, a processor 51 and memory 52, a processor 51 and auxiliary storage device 53, or a processing circuit 58 instead of a processor 51, memory 52 and auxiliary storage device 53. The processing circuit 58 is hardware that realizes at least a part of each part of each device. The processing circuit 58 may be dedicated hardware, or it may be a processor that executes a program stored in memory 52.

[0052] When the processing circuit 58 is dedicated hardware, specific examples of the processing circuit 58 include a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. Each device may have multiple processing circuits that substitute for the processing circuit 58. The multiple processing circuits share the role of the processing circuit 58.

[0053] In each device, some functions may be implemented by dedicated hardware, while the remaining functions may be implemented by software or firmware.

[0054] The processing circuit 58 can be implemented, in specific examples, by hardware, software, firmware, or a combination thereof. The processor 51, memory 52, auxiliary storage device 53, and processing circuit 58 are collectively referred to as the "processing circuitry." In other words, the function of each functional component of each device is realized by the processing circuitry.

[0055] Embodiment 2. The following will mainly describe the differences from the previously described embodiment, with reference to the drawings.

[0056] ***Configuration Description*** Figure 9 shows an example of the configuration of the input image generation device 10 according to this embodiment. The simulated image generation unit 101 according to this embodiment takes first ground surface information as input and generates a first simulated image using the trained model 211 and the first ground surface information. In this case, the simulated image generation unit 101 may also use information indicating the first imaging conditions as an auxiliary. The simulated image generation unit 101 also takes second ground surface information as input and generates a second simulated image using the trained model 212 and the second ground surface information. In this case, the simulated image generation unit 101 may also use information indicating the second imaging conditions as an auxiliary. At least one of the first ground surface information and the second ground surface information according to this embodiment is an actually acquired remote sensing image. The first ground surface information is, as a specific example, the radar image DIN6. The second ground surface information is, as a specific example, the optical image DIN7. The radar image DIN6 is a remote sensing image corresponding to the second domain. The optical image DIN7 is a remote sensing image corresponding to the first domain.

[0057] The trained model 211 is a model that has learned the relationship between a remote sensing image corresponding to the second domain and a remote sensing image corresponding to the first domain, and takes first ground surface information as input to infer the first ground surface information and the remote sensing image corresponding to the first domain. The trained model 211 may also be a model that has learned the relationship between a remote sensing image corresponding to the second domain and information indicating the first imaging conditions and a remote sensing image corresponding to the first domain, and takes first ground surface information and first imaging conditions as input to infer the remote sensing image. The trained model 212 is a model that has learned the relationship between a remote sensing image corresponding to the first domain and a remote sensing image corresponding to the second domain, and takes second ground surface information as input to infer the second ground surface information and the remote sensing image corresponding to the second domain. The trained model 212 may also be a model that has learned the relationship between a remote sensing image corresponding to the first domain and information indicating the second imaging conditions and a remote sensing image corresponding to the second domain, and takes second ground surface information and second imaging conditions as input to infer the remote sensing image. Each of the trained model 211 and trained model 212 may be a pre-prepared model, or it may be a model generated in the same manner as the inference models in embodiments 1 to 3 of Patent Document 1. Each of the trained model 211 and trained model 212 may be a model that learns the positional shift that occurs between a remote sensing image corresponding to the first domain and a remote sensing image corresponding to the second domain, and reflects the learned positional shift in its output. The functions of trained model 211 and trained model 212 may be realized by a single inference model.

[0058] The trained model storage unit 204 according to this embodiment further stores trained model 211 and trained model 212.

[0059] ***Explanation of Operation*** The operation of the alignment system 1 is the same as the operation of the alignment system 1 according to Embodiment 1. The differences between Embodiment 1 and Embodiment 2 will be mainly explained below.

[0060] (Step S101) The simulated image generation unit 101 takes the radar image DIN6 as input and generates a simulated optical satellite image D101A using the trained model 211 and the radar image DIN6. The simulated image generation unit 101 takes the optical image DIN7 as input and generates a simulated radar satellite image D101B using the trained model 212 and the optical image DIN7.

[0061] ***Explanation of the effects of Embodiment 2*** According to this embodiment, it is possible to simulate and generate a remote sensing image that corresponds to a domain different from the domain of the remote sensing image, using a remote sensing image that has actually been acquired.

[0062] ***Other Embodiments*** The embodiments described above can be freely combined, any component of each embodiment can be modified, or any component can be omitted in each embodiment. Furthermore, the embodiments are not limited to those shown in Embodiments 1 and 2, and various modifications can be made as needed. The procedures described using flowcharts, etc., may be modified as appropriate.

[0063] The various aspects of this disclosure are summarized below as an appendix.

[0064] (Note 1) An alignment system comprising: an input image generation device having a simulated image generation unit that simulates generating a remote sensing image as a first simulated image that shows the target area and corresponds to a first domain based on first ground surface information showing the ground surface in the target area, and a second simulated image that shows the target area and corresponds to a second domain different from the first domain based on second ground surface information showing the ground surface in the target area; and an image deformation unit that deforms at least one of the first simulated image and the second simulated image based on the positional displacement amount when a positional displacement amount indicating the amount of positional displacement set between the first simulated image and the second simulated image is generated.

[0065] (Note 2) The learning device further includes a model generation unit that generates a trained model that infers the positional displacement corresponding to the first inference image and the second inference image from a first inference image which is a remote sensing image whose corresponding domain is the first domain and a second inference image which is a remote sensing image whose corresponding domain is the second domain, by learning the relationship between the first learning image and the positional displacement using training data consisting of a first learning image, a second learning image and the positional displacement amount, and when the first simulated image is deformed, the first learning image is the first simulated image when the first simulated image is not deformed, the first learning image is the first simulated image when the second simulated image is deformed, and the alignment system as described in Note 1.

[0066] (Note 3) The alignment system according to Note 1 or 2, wherein the input image generation device further comprises a positional displacement amount generation unit that generates the positional displacement amount.

[0067] (Note 4) The alignment system according to Note 1 or 2, further comprising a positional displacement amount generating unit that generates the positional displacement amount, which is the learning device.

[0068] (Note 5) The alignment system according to any one of Notes 1 to 4, wherein the image deformation unit further deforms at least one of the first simulated image and the second simulated image, assuming that the ground surface changes in the target range between two time periods, such that one of the first simulated image and the second simulated image becomes a remote sensing image showing the ground surface before the change, and the other of the first simulated image and the second simulated image becomes a remote sensing image showing the ground surface after the change.

[0069] (Note 6) The alignment system according to any one of Notes 1 to 5, wherein the image deformation unit deforms at least one of the first simulated image and the second simulated image, assuming that disturbances occur when capturing remote sensing images, and further reflects the effect of disturbances on at least one of the first simulated image and the second simulated image.

[0070] (Note 7) The alignment system according to any one of Notes 1 to 6, wherein at least one of the first ground surface information and the second ground surface information is model information indicating a model of the ground surface in the target range, and the simulated image generation unit further generates the first simulated image based on information indicating first imaging conditions which are imaging conditions for a remote sensing image corresponding to the first domain when the first ground surface information is the model information, and further generates the second simulated image based on information indicating second imaging conditions which are imaging conditions for a remote sensing image corresponding to the second domain when the second ground surface information is the model information.

[0071] (Note 8) The alignment system according to any one of Notes 1 to 7, wherein the simulated image generation unit generates the first simulated image and the second simulated image using Image Translation AI (Artificial Intelligence).

[0072] (Note 9) The alignment system according to any one of Notes 1 to 8, wherein the first ground surface information is a target optical image which is an optical image indicating the target range, the first domain is a domain of the optical image, and the simulated image generation unit generates the first simulated image from the target optical image by adjusting each parameter of the target optical image.

[0073] (Note 10) The alignment system according to any one of Notes 1 to 9, wherein the second ground surface information is model information indicating a model of the ground surface in the target range, the second domain is a domain of a synthetic aperture radar image, and the simulated image generation unit generates the second simulated image using an RCS (Radar Cross-Section) simulator based on the model information.

[0074] (Note 11) The alignment system according to any one of Notes 1 to 6, wherein the first ground surface information is a remote sensing image corresponding to the second domain, and the simulated image generation unit generates the first simulated image using a trained model which is a model that has learned the relationship between the remote sensing image corresponding to the second domain and the remote sensing image corresponding to the first domain, and which takes the first ground surface information as input and infers the remote sensing image corresponding to the first ground surface information and the first domain, and the first ground surface information.

[0075] (Note 12) The alignment system according to any one of Notes 1 to 6, wherein the second ground surface information is a remote sensing image corresponding to the first domain, and the simulated image generation unit is a model that has learned the relationship between the remote sensing image corresponding to the first domain and the remote sensing image corresponding to the second domain, and takes the second ground surface information as input to infer the remote sensing image corresponding to the second ground surface information and the second domain, and generates the second simulated image using the second ground surface information and the trained model.

[0076] (Note 13) The alignment system according to any one of Notes 1 to 12, wherein the first domain is the domain of an optical image and the second domain is the domain of a synthetic aperture radar image.

[0077] 1 Alignment system, 10 Input image generation device, 101 Simulated image generation unit, 20 Learning device, 201 Positional displacement amount generation unit, 202 Image deformation unit, 203 Model generation unit, 204 Learned model storage unit, 210, 211, 212 Learned models, 30 Inference device, 301 Alignment unit, 51 Processor, 52 Memory, 53 Auxiliary storage device, 54 Input / Output IF, 55 Communication device, 58 Processing circuit, 59 Signal line, DIN1 Model information, DIN2 Optical satellite information, DIN3 Radar satellite information, DIN4, DIN7 Optical image, DIN5, DIN6 Radar image, D101A Optical satellite simulated image, D101B Radar satellite simulated image, DOUT1 Processed optical image, DOUT2 Processed radar image.

Claims

1. An alignment system comprising: an input image generation device having a simulated image generation unit that simulates generating a remote sensing image as a first simulated image that shows the target area and corresponds to a first domain based on first ground surface information showing the ground surface in the target area, and a second simulated image that shows the target area and corresponds to a second domain different from the first domain based on second ground surface information showing the ground surface in the target area; and an image deformation unit that deforms at least one of the first simulated image and the second simulated image based on the positional displacement amount when a positional displacement amount indicating the amount of positional displacement set between the first simulated image and the second simulated image is generated.

2. The learning device further comprises a model generation unit that uses learning data consisting of a first learning image, a second learning image, and the positional displacement amount to learn the relationship between the first learning image, the second learning image, and the positional displacement amount, thereby generating a trained model that infers the positional displacement amount corresponding to the first inference image and the second inference image from a first inference image which is a remote sensing image whose corresponding domain is the first domain and a second inference image which is a remote sensing image whose corresponding domain is the second domain, wherein when the first simulated image is deformed, the first learning image is the deformed first simulated image; when the first simulated image is not deformed, the first learning image is the first simulated image; when the second simulated image is deformed, the second learning image is the deformed second simulated image; and when the second simulated image is not deformed, the second learning image is the second simulated image.

3. The alignment system according to claim 1 or 2, wherein the input image generation device further comprises a positional displacement amount generation unit that generates the positional displacement amount.

4. The alignment system according to claim 1 or 2, further comprising a positional misalignment amount generation unit that generates the positional misalignment amount, wherein the learning device further comprises a positional misalignment amount generation unit that generates the positional misalignment amount.

5. The alignment system according to any one of claims 1 to 4, wherein the image deformation unit further deforms at least one of the first simulated image and the second simulated image, assuming that the ground surface changes in the target range between two time periods, such that one of the first simulated image and the second simulated image becomes a remote sensing image showing the ground surface before the change, and the other of the first simulated image and the second simulated image becomes a remote sensing image showing the ground surface after the change.

6. The alignment system according to any one of claims 1 to 5, wherein the image deformation unit deforms at least one of the first simulated image and the second simulated image, assuming that disturbances occur when capturing remote sensing images, thereby further reflecting the effect of disturbances on at least one of the first simulated image and the second simulated image.

7. The alignment system according to any one of claims 1 to 6, wherein at least one of the first ground surface information and the second ground surface information is model information indicating a model of the ground surface in the target range, and the simulated image generation unit, when the first ground surface information is the model information, further generates the first simulated image based on information indicating first imaging conditions which are imaging conditions for a remote sensing image corresponding to the first domain, and when the second ground surface information is the model information, further generates the second simulated image based on information indicating second imaging conditions which are imaging conditions for a remote sensing image corresponding to the second domain.

8. The alignment system according to any one of claims 1 to 7, wherein the simulated image generation unit generates the first simulated image and the second simulated image using Image Translation AI (Artificial Intelligence).

9. The alignment system according to any one of claims 1 to 8, wherein the first ground surface information is a target optical image which is an optical image indicating the target range, the first domain is a domain of the optical image, and the simulated image generation unit generates a first simulated image from the target optical image by adjusting each parameter of the target optical image.

10. The alignment system according to any one of claims 1 to 9, wherein the second ground surface information is model information indicating a model of the ground surface in the target range, the second domain is a domain of a synthetic aperture radar image, and the simulated image generation unit generates the second simulated image using an RCS (Radar Cross-Section) simulator based on the model information.

11. The alignment system according to any one of claims 1 to 6, wherein the first ground surface information is a remote sensing image corresponding to the second domain, and the simulated image generation unit generates the first simulated image using a trained model which is a model that has learned the relationship between the remote sensing image corresponding to the second domain and the remote sensing image corresponding to the first domain, and which takes the first ground surface information as input and infers the remote sensing image corresponding to the first ground surface information and the first domain, and the first ground surface information.

12. The alignment system according to any one of claims 1 to 6, wherein the second ground surface information is a remote sensing image corresponding to the first domain, and the simulated image generation unit generates the second simulated image using a trained model which is a model that has learned the relationship between the remote sensing image corresponding to the first domain and the remote sensing image corresponding to the second domain, and which takes the second ground surface information as input and infers the remote sensing image corresponding to the second ground surface information and the second domain, and the second ground surface information.

13. The alignment system according to any one of claims 1 to 12, wherein the first domain is an optical image domain and the second domain is a synthetic aperture radar image domain.

14. A computer-based input image generation device that simulates generating a remote sensing image corresponding to a first domain as a first simulated image, based on first ground surface information indicating the ground surface in the target area; a computer-based learning device that, when a positional displacement amount indicating the amount of positional displacement set between the first simulated image and the second simulated image is generated, deforms at least one of the first simulated image and the second simulated image based on the positional displacement amount.

15. A positioning program that causes a computer, which is an input image generation device, to perform a simulated image generation process, which simulates generating a remote sensing image that shows the target area and corresponds to a first domain as a first simulated image based on first ground surface information showing the ground surface in the target area, and simulates generating a remote sensing image that shows the target area and corresponds to a second domain different from the first domain as a second simulated image based on second ground surface information showing the ground surface in the target area, and causes a computer, which is a learning device, to perform an image deformation process, which deforms at least one of the first simulated image and the second simulated image, based on the position deformation amount, when a positional displacement amount indicating the amount of positional displacement set between the first simulated image and the second simulated image is generated.

Images