Alignment system, alignment method, and alignment program

Abstract

The registration system (1) includes an input image generation device (10) and a learning device (20). The input image generation device (10) includes a simulated image generation unit that generates a first simulated image that indicates the target range and corresponds to a first domain based on first ground surface information that indicates the ground surface in the target range, and generates a second simulated image that indicates the target range and corresponds to a second domain based on second ground surface information that indicates the ground surface in the target range. The learning device (20) includes an image deformation unit that deforms at least one of the first simulated image and the second simulated image based on an amount of positional deviation set between the first simulated image and the second simulated image.

Description

[Technical Field]

[0001] The present disclosure relates to a registration system, a registration method, and a registration program. [Background technology]

[0002] When using multiple satellite images that correspond to different domains, alignment is required. Patent Document 1 discloses a technique for performing registration between SAR images and optical images using an inference model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7262679 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology disclosed in Patent Document 1 utilizes actually acquired remote sensing images. However, in the field of disaster prevention, there are realistic scenarios in which remote sensing images corresponding to the desired domain are not actually acquired, such as optical images stored in archives and SAR (Synthetic Aperture Radar) images acquired immediately after a disaster. The present disclosure relates to a technology for aligning remote sensing images whose corresponding domains are different from each other, and aims to construct an inference model that can be used to perform alignment when the remote sensing images have not actually been acquired. [Means for solving the problem]

[0005] The alignment system according to the present disclosure comprises: generating a simulated remote sensing image, which indicates the target range and corresponds to the first domain, as a first simulated image based on first ground surface information indicating the ground surface in the target range; a simulated image generating unit that generates, as a second simulated image, a remote sensing image that indicates the target range and corresponds to a second domain different from the first domain, based on second ground surface information that indicates the ground surface in the target range; an input image generation device comprising: an image deformation unit that, when a positional deviation amount indicating an amount of positional deviation to be set between the first simulated image and the second simulated image is generated, deforms at least one of the first simulated image and the second simulated image based on the positional deviation amount; A learning device comprising: Equipped with. [Effects of the Invention]

[0006] According to the present disclosure, a simulated image generation unit of an input image generation device generates two types of simulated images corresponding to two types of domains, respectively. An image transformation unit of a learning device transforms at least one of the two types of simulated images based on the generated misalignment amount. Here, an inference model that can be used for alignment can be constructed using learning data consisting of the two transformed simulated images and the misalignment amount. Therefore, by utilizing the present disclosure, it is possible to realize a technique for performing registration between remote sensing images having different corresponding domains. Inferential models can be constructed that are utilized to perform registration when not available. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a diagram showing an example of the configuration of an alignment system 1 according to a first embodiment. [Figure 2] 1 is a diagram showing an example of the configuration of an input image generating device 10 according to a first embodiment. [Figure 3] FIG. 2 is a diagram showing an example of the configuration of a learning device 20 according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of the configuration of an inference device 30 according to the first embodiment. [Figure 5] FIG. 2 is a diagram showing an example of the hardware configuration of each device according to the first embodiment. [Figure 6] 5 is a flowchart showing the operation of the alignment system 1 in the learning phase according to the first embodiment. [Figure 7] 5 is a flowchart showing the operation of the alignment system 1 in the inference phase according to the first embodiment. [Figure 8] FIG. 10 is a diagram showing an example of the hardware configuration of each device according to a modification of the first embodiment. [Figure 9] FIG. 10 is a diagram showing an example of the configuration of an input image generating device 10 according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] In the description of the embodiments and drawings, the same elements and corresponding elements are given the same symbols. Descriptions of elements given the same symbols are omitted or simplified as appropriate. Arrows in the drawings mainly indicate the flow of data or the flow of processing. Furthermore, "unit" may be interpreted as "circuit," "device," "equipment," "process," "step," "procedure," "processing," or "circuitry" as appropriate. The functions of each unit in each device may be realized by firmware, software, hardware, or a combination of these.

[0009] Embodiment 1 Hereinafter, this embodiment will be described in detail with reference to the drawings.

[0010] ***Configuration Description*** Fig. 1 shows an example of the configuration of a registration system 1 according to this embodiment. As shown in Fig. 1, the registration system 1 includes an input image generation device 10, a learning device 20, an inference device 30, and a trained model storage unit 204. The multiple devices included in the registration system 1 may be configured integrally as appropriate. In the following, a combination of an optical satellite image and a radar satellite image will be described as an example of a combination of remote sensing images that correspond to different domains. However, this embodiment may also be applied to combinations of other types of remote sensing images. Specific examples of remote sensing images include optical images, SAR (Synthetic Aperture Radar) images, and infrared images. Remote sensing images may be images captured from an aircraft or satellites.

[0011] 2 shows an example of the configuration of the input image generation device 10. The input image generation device 10 includes a simulated image generation unit 101 as shown in FIG. The simulated image generation unit 101 generates a simulated remote sensing image showing a target range and corresponding to a first domain as a first simulated image based on the first ground surface information. The simulated image generation unit 101 generates a simulated remote sensing image showing a target range and corresponding to a second domain as a second simulated image based on the second ground surface information. The simulated image generation unit 101 uses a simulation technique when generating the simulated image. The target range is any range on the earth's surface. The target range may be set in any manner and may be changed as appropriate. Each of the first ground surface information and the second ground surface information is information indicating the ground surface in the target range. At least one of the first ground surface information and the second ground surface information is information indicating the ground surface in the target range. It may be model information indicating a model. "Simulated image" is a general term for various types of simulated remote sensing images. The information about the remote sensing image may include information about the sensor that captures the remote sensing image, and may also include information about the aircraft or satellite that is equipped with the sensor. When the first ground surface information is model information, the simulated image generating unit 101 may generate a first simulated image based on information indicating a first imaging condition. When the second ground surface information is model information, the simulated image generating unit 101 may generate a second simulated image based on information indicating a second imaging condition. The first imaging condition is an imaging condition for a remote sensing image corresponding to the first domain. The second imaging condition is an imaging condition for a remote sensing image corresponding to the second domain. The first domain is illustratively the domain of the optical image. The second domain is a domain different from the first domain, and is specifically a domain of a synthetic aperture radar image. As a specific example, the simulated image generation unit 101 receives model information DIN1, optical satellite information DIN2, and radar satellite information DIN3 as input, generates an optical satellite simulated image D101A based on the model information DIN1 and the optical satellite information DIN2, and generates a radar satellite simulated image D101B based on the model information DIN1 and the radar satellite information DIN3. The simulated image generation unit 101 then outputs the optical satellite simulated image D101A and the radar satellite simulated image D101B. As a specific example, the simulated image generation unit 101 generates each simulated image using at least one of a method using Image Translation AI (Artificial Intelligence), a legacy method, and a method using an RCS (Radar Cross-Section) simulator from a model of the earth's surface. Specific examples of methods using Image Translation AI include refinement, GAN (Generative Adversarial Network), or GSG (Global Scene-harmonious Guidance). A specific example of a legacy method is a method of artificially adjusting at least one of the brightness value and the signal-to-noise (SN) value of a given optical image to generate a simulated optical image. Specific examples of a legacy method include at least one of a method of converting an optical image to a black-and-white image and adding noise to the black-and-white image, a method of adjusting the contrast of the optical image, and a method of inverting the color of the optical image. In a legacy method, a user may visually specify each parameter value related to the legacy method, and the simulated image generation unit 101 may set each parameter value according to the user's instructions. In other words, the simulated image generation unit 101 may generate a first simulated image from a target optical image, which is an optical image showing a target range, by adjusting each parameter of the target optical image. The simulated image generation unit 101 may also process the simulated image generated using the legacy method using AI and output the processed simulated image. The method using the RCS simulator is a method of simulating a SAR image using the RCS simulator based on the model information DIN1.

[0012] The model information DIN1 is information indicating a model of the shape of the earth's surface. Specific examples of the model information DIN1 include a DEM (Digital Elevation Model), a DSM (Digital Surface Model), or a three-dimensional model (such as a blender). The simulated image generation unit 101 may use only the model corresponding to the target range among the models indicated by the model information DIN1.

[0013] The optical satellite information DIN2 is information indicating the imaging conditions of the optical satellite simulated image D101A. Specific examples of the imaging conditions include imaging parameters related to the optical sensor mounted on the target optical satellite, the first imaging time point, and external factors of the optical satellite at the first imaging time point. and information indicating the The imaging parameters indicated by the optical satellite information DIN2 include, for example, parameters related to the band (spectrum), resolution, observation mode, brightness, observation angle, pointing angle (also called off-nadir angle), and imaging time. In the case of TDI (Telephoto Integration) imaging, the imaging time is the total integration time (the product of the number of TDI stages and the imaging period) which corresponds to the exposure time in a general camera, and the number of TDI stages is set in the imaging parameters. The first imaging time point is a virtual imaging time point set as the imaging time point of the first simulated image. Specific examples of external factors of the optical satellite at the time of the first image capture include the season at the time of the first image capture, the solar altitude at the time of the first image capture, and the weather and atmospheric conditions around the target area at the time of the first image capture.

[0014] The radar satellite information DIN3 is information indicating the imaging conditions of the radar satellite simulated image D101B. Specifically, the imaging conditions include information indicating imaging parameters for the radar mounted on the target radar satellite, the second imaging time point, and information indicating external factors at the second imaging time point. Specifically, the radar is a SAR. The imaging parameters indicated by the radar satellite information DIN3 include, for example, parameters related to resolution, observation mode, observation angle, incidence angle, imaging azimuth angle, and radio wave irradiation time. Note that even for satellites with the same specifications, resolution can change depending on the pointing angle and incidence angle, etc. Here, the name of the satellite capturing the image and the imaging mode each correspond to imaging parameters. This is because the specifications are determined according to the satellite's name. As a specific example, a satellite called "ALOS-2" uses L-band radio waves, and the approximate range of resolution is determined according to the imaging mode used, with the detailed resolution determined further by information such as the angle of incidence. The second imaging time point is a virtual imaging time point set as the imaging time point of the second simulated image. Note that the first imaging time point and the second imaging time point may be different from each other. A specific example of each external factor at the time of the second imaging includes the orbit of the radar satellite at the time of the second imaging.

[0015] The simulated optical satellite image D101A corresponds to the first simulated image. The simulated optical satellite image D101A is a simulated optical image and corresponds to a remote sensing image captured at a first capture time.

[0016] The radar satellite simulated image D101B corresponds to the second simulated image. The radar satellite simulated image D101B is a simulated radar image, and corresponds to a remote sensing image captured at the second imaging time point.

[0017] 3 shows an example of the configuration of the learning device 20. As shown in FIG. 3, the learning device 20 includes a positional deviation amount generating unit 201, an image deformation unit 202, and a model generating unit 203.

[0018] The positional shift amount generating unit 201 generates the positional shift amount between two types of remote sensing images as the positional shift amount. Note that the input image generating device 10 may include the positional shift amount generating unit 201 instead of the learning device 20. As a specific example, the positional deviation amount generating unit 201 generates a positional deviation amount for generating a positional deviation between the simulated optical satellite image D101A and the simulated radar satellite image D101B.

[0019] The positional deviation amount indicates the amount of positional deviation set between the first simulated image and the second simulated image, and corresponds to the amount of displacement between the first simulated image and the second simulated image. The positional deviation amount may be a scalar value. The positional deviation amount may be a random value, a value representing a positional deviation caused by insufficient precision of the small satellite, or a value generated according to the characteristics of the Earth's surface. The positional deviation amount may be a value that represents the displacement of pixels generated by the ground surface. Specific examples of the ground surface features include the presence of artificial objects such as buildings, or the presence of unevenness related to the terrain (such as the presence of mountains or valleys). In other words, the positional deviation amount generating unit 201 may calculate the positional deviation amount based on a value based on DEM or DSM, or a three-dimensional model, taking into account factors such as collapse caused by altitude. Specific examples of the positional deviation amount generating unit 201 may set different values ​​for the positional deviation amount in mountainous areas and the positional deviation amount in flat areas.

[0020] The image deformation unit 202 deforms at least one of the first simulated image and the second simulated image based on the displacement amount generated by the displacement amount generation unit 201, thereby generating a simulated pair of remote sensing images with a known displacement amount. As a specific example, the image deformation unit 202 receives at least one of an optical satellite simulated image D101A and a radar satellite simulated image D101B as input, and deforms the input simulated image based on the positional shift amount generated by the positional shift amount generation unit 201.

[0021] The image deformation unit 202 may further deform at least one of the first simulated image and the second simulated image so that, assuming that the ground surface changes in the target range between two time periods, one of the first simulated image and the second simulated image becomes a remote sensing image showing the ground surface before the change, and the other of the first simulated image and the second simulated image becomes a remote sensing image showing the ground surface after the change. In other words, the image deformation unit 202 may further deform at least one of the simulated images for the purpose of generating a pair of remote sensing images corresponding to before and after a change in the ground surface that occurs in the target range between two time periods as an image pair for two time periods. Specific examples of changes in the earth's surface include at least one of the following: changes in agricultural land or green space due to seasonal changes; changes in man-made objects due to the construction or demolition of buildings; changes in water levels due to flooding; changes in relief due to volcanic ejecta or landslides; and changes in topography due to earthquakes. A two-time image pair is a pair of a remote sensing image taken at a certain time and a remote sensing image taken at a time different from the certain time. As a specific example, the certain time is a period before the disaster occurs, and the other time is a period after the disaster occurs. As another specific example, the certain time is a certain season, and the other time is a season different from the certain season. Furthermore, the image deformation unit 202 may add noise caused by the influence of a disturbance to at least one of the simulated images for the purpose of learning the influence of the disturbance. In other words, assuming that a disturbance occurs when capturing a remote sensing image, the image deformation unit 202 may further reflect the influence of the disturbance in at least one of the first simulated image and the second simulated image by deforming at least one of the first simulated image and the second simulated image. Note that the disturbance corresponding to each simulated image does not need to be constant.

[0022] The model generation unit 203 generates a trained model 210 by learning the relationship between the first training image, the second training image, and the amount of misalignment using training data consisting of the first training image, the second training image, and the amount of misalignment. Here, when the first simulated image is deformed, the first training image is the first simulated image after deformation. When the first simulated image is not deformed, the first training image is the first simulated image. When the second simulated image is deformed, the second training image is the second simulated image after deformation. When the second simulated image is not deformed, the second training image is the second simulated image. As a specific example, the model generation unit 203 generates the trained model 210 by learning the relationship between the optical satellite simulated image D101A and the radar satellite simulated image D101B and the amount of positional deviation occurring between the optical satellite simulated image D101A and the radar satellite simulated image D101B. The amount of positional deviation corresponds to the correct answer. In this case, the model generation unit 203 may utilize machine learning or AI (Artificial Intelligence) technology, and may generate the trained model 210 in the same way as the inference model of embodiment 4 of Patent Document 1. Model The generation unit 203 may generate the trained model 210 by semi-supervised learning or reinforcement learning. The model generation unit 203 stores the generated trained model 210 in the trained model storage unit 204.

[0023] The trained model storage unit 204 stores the trained model 210.

[0024] The trained model 210 is a model that infers the amount of misalignment between two remote sensing images that correspond to different domains. That is, the trained model 210 is a model that infers the amount of misalignment corresponding to a first inference-use image and a second inference-use image from a first inference-use image that is a remote sensing image whose corresponding domain is a first domain and a second inference-use image that is a remote sensing image whose corresponding domain is a second domain. As a specific example, the trained model 210 is a model that infers the amount of misalignment between a remote sensing image whose corresponding domain is an optical image and a remote sensing image whose corresponding domain is an SAR image. The amount of misalignment inferred by the trained model 210 is used when aligning the two remote sensing images. The trained model 210 may be an inference model specialized for imaging conditions, or may be an inference model with high versatility. An inference model specialized for imaging conditions is a model generated by using training data corresponding to the same or similar imaging conditions, and can infer the amount of misalignment with a relatively high degree of accuracy when inferring the amount of misalignment between two remote sensing images corresponding to imaging conditions that are the same or similar to the imaging conditions corresponding to the training data of the inference model. A versatile inference model is a model generated by using training data corresponding to various imaging conditions. The inference model may be a model that further learns the relationship between each parameter corresponding to the imaging condition and the amount of misalignment, or may be a model that infers the amount of misalignment by auxiliary use of each parameter corresponding to the imaging condition.

[0025] 4 shows an example of the configuration of the inference device 30. The inference device 30 includes a positioning unit 301. The registration unit 301 infers the amount of misalignment between two remote sensing images corresponding to different domains by inputting the two remote sensing images to the trained model 210. Then, the registration unit 301 performs registration between the two remote sensing images based on the inferred amount of misalignment. The two remote sensing images may be images from OpenEarthMap, or may be images taken at different times, such as before and after the disaster. As a specific example, the alignment unit 301 receives the optical image DIN4 and the radar image DIN5 as input, inputs the input optical image DIN4 and radar image DIN5 to the trained model 210, and acquires the amount of misalignment corresponding to the optical image DIN4 and the radar image DIN5 from the trained model 210. Thereafter, the alignment unit 301 aligns the optical image DIN4 and the radar image DIN5 based on the acquired amount of misalignment, and outputs a processed optical image DOUT1 and a processed radar image DOUT2. Land cover classification may be performed on each registered remote sensing image.

[0026] The optical image DIN4 is a remote sensing image taken by a target optical satellite.

[0027] The radar image DIN5 is a remote sensing image taken by a radar satellite of the target.

[0028] The processed optical image DOUT1 is an image corresponding to the optical image DIN4, and corresponds to the result of aligning the optical image DIN4 with the radar image DIN5. The processed optical image DOUT1 may be the same as the optical image DIN4, or may be an image obtained by processing the optical image DIN4.

[0029] The processed radar image DOUT2 is an image corresponding to the radar image DIN5, and corresponds to the result of aligning the optical image DIN4 and the radar image DIN5. The processed radar image DOUT2 may be the same as the radar image DIN5, or may be an image obtained by processing the radar image DIN5.

[0030] 5 shows an example of the hardware configuration of each device according to this embodiment. Each device is made up of a computer. Each device may be made up of multiple computers.

[0031] As shown in the figure, each device is a computer equipped with hardware such as a processor 51, a memory 52, an auxiliary storage device 53, an input / output IF (Interface) 54, and a communication device 55. These pieces of hardware are appropriately connected via signal lines 59.

[0032] The processor 51 is an integrated circuit (IC) that performs arithmetic processing and controls the hardware of the computer. Specific examples of the processor 51 include a central processing unit (CPU), a digital signal processor (DSP), or a graphics processing unit (GPU). Each device may include multiple processors that take the place of processor 51. The multiple processors share the role of processor 51.

[0033] The memory 52 is typically a volatile storage device, specifically a random access memory (RAM). The memory 52 is also called a primary storage device or a main memory. Data stored in the memory 52 is saved in the secondary storage device 53 as needed.

[0034] The auxiliary storage device 53 is typically a non-volatile storage device, and specific examples thereof include a ROM (Read Only Memory), an HDD (Hard Disk Drive), or a flash memory. Data stored in the auxiliary storage device 53 is loaded into the memory 52 as needed. The memory 52 and the auxiliary storage device 53 may be integrated into one unit.

[0035] The input / output IF 54 is a port to which an input device and an output device are connected. A specific example of the input / output IF 54 is a USB (Universal Serial Bus) terminal. Specific examples of the input device are a keyboard and a mouse. A specific example of the output device is a display.

[0036] The communication device 55 is a receiver and a transmitter, and is specifically a communication chip or a network interface card (NIC).

[0037] Each unit of each device may use the input / output IF 54 and the communication device 55 as appropriate when communicating with other devices.

[0038] The auxiliary storage device 53 stores a positioning program. The registration program is a program that causes a computer to realize the functions of the respective units of the respective devices. The registration program is loaded into the memory 52 and executed by the processor 51.

[0039] Data used when executing the alignment program and data obtained by executing the alignment program are stored in a storage device as appropriate. Each part of each device uses a storage device as appropriate. As a specific example, the storage device comprises at least one of the memory 52, the auxiliary storage device 53, a register in the processor 51, and a cache memory in the processor 51. Note that the terms "data" and "information" may have the same meaning. The storage device may be independent of the computer. The functions of the memory 52 and the auxiliary storage device 53 may be realized by other storage devices.

[0040] The alignment program may be recorded on a computer-readable non-volatile recording medium. Specific examples of the non-volatile recording medium include an optical disk and a flash memory. The alignment program may be provided as a program product.

[0041] ***Explanation of Operation*** The operating procedures of the devices that make up the alignment system 1 are collectively called an alignment method. Also, the programs that realize the operations of the devices that make up the alignment system 1 are collectively called alignment programs.

[0042] 6 is a flowchart showing an example of the operation of the alignment system 1 in the learning phase. This operation will be explained using FIG.

[0043] (Step S101) The simulated image generating unit 101 receives model information DIN1, optical satellite information DIN2, and radar satellite information DIN3 as input, and generates an optical satellite simulated image D101A and a radar satellite simulated image D101B.

[0044] (Step S102) The positional deviation amount generating unit 201 generates a positional deviation amount corresponding to the positional deviation set between the optical satellite simulated image D101A and the radar satellite simulated image D101B.

[0045] (Step S103) The image deformation unit 202 appropriately deforms at least one of the optical satellite simulated image D101A and the radar satellite simulated image D101B based on the positional deviation amount generated by the positional deviation amount generation unit 201. The image transformation unit 202 may further transform at least one of the optical satellite simulated image D101A and the radar satellite simulated image D101B, taking into account at least one of changes in the Earth's surface and disturbances that occur between the two time periods.

[0046] (Step S104) The model generation unit 203 generates a trained model 210 using training data consisting of an optical satellite simulated image D101A, a radar satellite simulated image D101B, and positional deviation amounts corresponding to the optical satellite simulated image D101A and the radar satellite simulated image D101B. The model generation unit 203 stores the generated trained model 210 in the trained model storage unit 204.

[0047] 7 is a flowchart showing an example of the operation of the registration system 1 in the inference phase. This operation will be explained using FIG.

[0048] (Step S111) The alignment unit 301 acquires the trained model 210 from the trained model storage unit 204 and infers the amount of positional deviation between the optical image DIN4 and the radar image DIN5 by inputting the optical image DIN4 and the radar image DIN5 into the trained model 210. Thereafter, the alignment unit 301 performs alignment between the optical image DIN4 and the radar image DIN5 based on the inferred amount of positional deviation, and outputs a processed optical image DOUT1 and a processed radar image DOUT2.

[0049] ***Explanation of the effect of the first embodiment*** In the prior art, if a sufficient number of remote sensing images were not actually acquired, it was not possible to align different types of remote sensing images, and as a result, it was not possible to compare analysis results between different types of remote sensing images. On the other hand, in this embodiment, a pair of different remote sensing images is simulated as a pair of simulated images, and an inference model is generated that infers the amount of positional shift between the different remote sensing images based on the generated pair of simulated images. Therefore, according to this embodiment, when a pair of different remote sensing images, such as a pair of an optical image and an SAR image, is given, by utilizing the inference model, it is possible to generate remote sensing images so that the amount of positional shift between each pixel in the images is relatively small.

[0050] Furthermore, in this embodiment, instead of preparing a pair of images at two different times by actually capturing remote sensing images, simulated images before and after deformation can be used as a pair of images at two different times. Therefore, this embodiment has the advantage of increasing the amount of data sets that can be secured. Furthermore, according to this embodiment, in order to perform inference taking into account changes in the ground surface or disturbances during two periods, an inference model can be generated by appropriately generating a simulated image other than the simulated image after the change and the simulated image after the change as training data. By utilizing this inference model, it becomes possible to estimate the positional deviation amount with a relatively high degree of accuracy.

[0051] ***Other Configurations*** <Variation 1> FIG. 8 shows an example of the hardware configuration of each device according to this modification. Each device includes a processing circuit 58 in place of the processor 51 , the processor 51 and memory 52 , the processor 51 and auxiliary storage device 53 , or the processor 51 , memory 52 and auxiliary storage device 53 . The processing circuitry 58 is hardware that realizes at least a part of each unit provided in each device. The processing circuitry 58 may be dedicated hardware, or may be a processor that executes a program stored in the memory 52 .

[0052] When processing circuitry 58 is dedicated hardware, processing circuitry 58 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. Each device may include multiple processing circuits that replace processing circuit 58. The multiple processing circuits share the role of processing circuit 58.

[0053] In each device, some functions may be realized by dedicated hardware, and the remaining functions may be realized by software or firmware.

[0054] Processing circuitry 58 is illustratively implemented in hardware, software, firmware, or a combination thereof. The processor 51, memory 52, auxiliary storage device 53, and processing circuit 58 are collectively referred to as "processing circuitry." In other words, the functions of the functional components of each device are realized by the processing circuitry.

[0055] Embodiment 2 The following mainly describes the differences from the above-described embodiment with reference to the drawings.

[0056] ***Configuration Description*** FIG. 9 shows an example of the configuration of an input image generating device 10 according to this embodiment. The simulated image generating unit 101 according to this embodiment receives the first ground surface information as an input and generates a first simulated image using the trained model 211 and the first ground surface information. At this time, the simulated image generating unit 101 may supplementarily use information indicating the first imaging condition. Furthermore, the simulated image generating unit 101 receives the second ground surface information as an input and generates a second simulated image using the trained model 212 and the second ground surface information. At this time, the simulated image generating unit 101 may supplementarily use information indicating the second imaging conditions. At least one of the first earth's surface information and the second earth's surface information according to this embodiment is an actually acquired remote sensing image. A specific example of the first ground surface information is a radar image DIN6. A specific example of the second earth's surface information is the optical image DIN7. The radar image DIN6 is a remote sensing image corresponding to the second domain. The optical image DIN7 is a remote sensing image corresponding to the first domain.

[0057] The trained model 211 is a model that has learned the relationship between a remote sensing image corresponding to the second domain and a remote sensing image corresponding to the first domain, and is a model that uses first ground surface information as input to infer the first ground surface information and the remote sensing image corresponding to the first domain. The trained model 211 may be a model that has learned the relationship between a remote sensing image corresponding to the second domain and information indicating the first imaging conditions, and the remote sensing image corresponding to the first domain, and is a model that uses the first ground surface information and the first imaging conditions as input to infer the remote sensing image. The trained model 212 is a model that has learned the relationship between a remote sensing image corresponding to a first domain and a remote sensing image corresponding to a second domain, and is a model that uses second ground surface information as input to infer the second ground surface information and the remote sensing image corresponding to the second domain. The trained model 212 may be a model that has learned the relationship between a remote sensing image corresponding to the first domain and information indicating second imaging conditions, and the remote sensing image corresponding to the second domain, and is a model that uses the second ground surface information and second imaging conditions as input to infer the remote sensing image. Each of trained model 211 and trained model 212 may be a model prepared in advance, or may be a model generated similarly to the inference models of embodiments 1 to 3 of Patent Document 1. Each of trained model 211 and trained model 212 may be a model that learns the positional shift that occurs between a remote sensing image corresponding to a first domain and a remote sensing image corresponding to a second domain, and reflects the learned positional shift in the output. The functions of trained model 211 and trained model 212 may be realized by a single inference model.

[0058] The trained model storage unit 204 according to this embodiment further stores a trained model 211 and a trained model 212.

[0059] ***Explanation of Operation*** The operation of the alignment system 1 is the same as that of the alignment system 1 according to embodiment 1. The following mainly describes the differences between embodiment 1 and embodiment 2.

[0060] (Step S101) The simulated image generation unit 101 receives the radar image DIN6 as input and generates an optical satellite simulated image D101A using the trained model 211 and the radar image DIN6. The simulated image generation unit 101 receives the optical image DIN7 as input and generates a radar satellite simulated image D101B using the trained model 212 and the optical image DIN7.

[0061] ***Explanation of the effect of the second embodiment*** According to this embodiment, it is possible to use an actually acquired remote sensing image to generate a simulated remote sensing image corresponding to a domain different from the domain of the remote sensing image.

[0062] ***Other embodiments*** The above-described embodiments may be freely combined, or any of the components in each embodiment may be modified, or any of the components in each embodiment may be omitted. Furthermore, the embodiments are not limited to those shown in Embodiments 1 and 2, and various modifications are possible as needed. The procedures explained using flowcharts and the like may be modified as appropriate.

[0063] Various aspects of the present disclosure are summarized below as appendices.

[0064] (Appendix 1) generating a simulated remote sensing image, which indicates the target range and corresponds to the first domain, as a first simulated image based on first ground surface information indicating the ground surface in the target range; a simulated image generating unit that generates, as a second simulated image, a remote sensing image that indicates the target range and corresponds to a second domain different from the first domain, based on second ground surface information that indicates the ground surface in the target range; an input image generation device comprising: an image deformation unit that, when a positional deviation amount indicating an amount of positional deviation to be set between the first simulated image and the second simulated image is generated, deforms at least one of the first simulated image and the second simulated image based on the positional deviation amount; A learning device comprising: An alignment system comprising:

[0065] (Appendix 2) The learning device further a model generation unit that uses training data including a first training image, a second training image, and the amount of misalignment to learn a relationship between the first training image and the second training image and the amount of misalignment, thereby generating a trained model that infers the amount of misalignment corresponding to the first inference image and the second inference image from the first inference image, which is a remote sensing image whose corresponding domain is the first domain, and the second inference image, which is a remote sensing image whose corresponding domain is the second domain. Equipped with When the first simulated image is transformed, the first learning image is the transformed first simulated image, When the first simulated image is not transformed, the first training image is the first simulated image; When the second simulated image is transformed, the second learning image is the second simulated image after the transformation, 2. The alignment system of claim 1, wherein the second training image is the second simulated image when the second simulated image is not deformed.

[0066] (Appendix 3) The input image generation device further comprises: a positional deviation amount generating unit for generating the positional deviation amount; 3. The alignment system of claim 1 or 2, comprising:

[0067] (Appendix 4) The learning device further a positional deviation amount generating unit for generating the positional deviation amount; 3. The alignment system of claim 1 or 2, comprising:

[0068] (Appendix 5) 5. The alignment system according to any one of appendices 1 to 4, wherein the image deformation unit further deforms at least one of the first simulated image and the second simulated image so that, assuming that the ground surface in the target range changes between two time periods, one of the first simulated image and the second simulated image becomes a remote sensing image showing the ground surface before the change, and the other of the first simulated image and the second simulated image becomes a remote sensing image showing the ground surface after the change.

[0069] (Appendix 6) 6. The alignment system according to any one of appendices 1 to 5, wherein, assuming that a disturbance occurs when capturing a remote sensing image, the image deformation unit deforms at least one of the first simulated image and the second simulated image to further reflect the influence of the disturbance on at least one of the first simulated image and the second simulated image.

[0070] (Appendix 7) At least one of the first ground surface information and the second ground surface information is model information indicating a model of the ground surface in the target range, The simulated image generation unit When the first earth's surface information is the model information, the first simulated image is generated based on information indicating a first imaging condition, which is an imaging condition of a remote sensing image corresponding to the first domain; An alignment system described in any one of Appendices 1 to 6, further generating the second simulated image based on information indicating second imaging conditions, which are imaging conditions of a remote sensing image corresponding to the second domain, when the second earth's surface information is the model information.

[0071] (Appendix 8) The alignment system according to any one of claims 1 to 7, wherein the simulated image generation unit generates the first simulated image and the second simulated image using Image Translation AI (Artificial Intelligence).

[0072] (Appendix 9) the first ground surface information is a target optical image that is an optical image showing the target range, the first domain is a domain of an optical image; 9. The position according to claim 1, wherein the simulated image generation unit generates the first simulated image from the target optical image by adjusting each parameter of the target optical image. Matching system.

[0073] (Appendix 10) the second ground surface information is model information indicating a model of the ground surface in the target range, the second domain is a domain of synthetic aperture radar images; 10. The alignment system according to claim 1, wherein the simulated image generation unit generates the second simulated image using a Radar Cross-Section (RCS) simulator based on the model information.

[0074] (Appendix 11) the first earth surface information is a remote sensing image corresponding to the second domain; The alignment system of any one of Appendices 1 to 6, wherein the simulated image generation unit generates the first simulated image using a trained model that has learned the relationship between a remote sensing image corresponding to the second domain and a remote sensing image corresponding to the first domain, and that uses the first earth surface information as input and infers the first earth surface information and the remote sensing image corresponding to the first domain.

[0075] (Appendix 12) the second earth surface information is a remote sensing image corresponding to the first domain; The alignment system of any one of Appendices 1 to 6, wherein the simulated image generation unit generates the second simulated image using a trained model that has learned the relationship between a remote sensing image corresponding to the first domain and a remote sensing image corresponding to the second domain, and that uses the second earth surface information as input and infers the second earth surface information and the remote sensing image corresponding to the second domain.

[0076] (Appendix 13) the first domain is a domain of an optical image; 13. The alignment system of any one of claims 1 to 12, wherein the second domain is a domain of a synthetic aperture radar image. [Explanation of symbols]

[0077] 1 Alignment system, 10 Input image generation device, 101 Simulated image generation unit, 20 Learning device, 201 Position deviation generation unit, 202 Image deformation unit, 203 Model generation unit, 204 Trained model memory unit, 210, 211, 212 Trained model, 30 Inference device, 301 Alignment unit, 51 Processor, 52 Memory, 53 Auxiliary storage device, 54 Input / output IF, 55 Communication device, 58 Processing circuit, 59 Signal line, DIN1 Model information, DIN2 Optical satellite information, DIN3 Radar satellite information, DIN4, DIN7 Optical image, DIN5, DIN6 Radar image, D101A Optical satellite simulated image, D101B Radar satellite simulated image, DOUT1 Processed optical image, DOUT2 Processed radar image.

Claims

1. A simulated image generating unit that generates a simulated first simulated image, which is a remote sensing image corresponding to the target range and a first domain, and a simulated second simulated image, which is a remote sensing image corresponding to the target range and a second domain different from the first domain, based on first ground surface information and second ground surface information that indicate the ground surface of the target range, respectively. an input image generation device comprising: an image deformation unit that, when a positional deviation amount indicating the amount of positional deviation set between the first simulated image and the second simulated image is generated, deforms at least one of the first simulated image and the second simulated image based on the positional deviation amount; a model generation unit that uses training data including a first training image, a second training image, and the displacement amount to learn a relationship between the first training image, the second training image, and the displacement amount, and that receives as input a remote sensing image whose corresponding domain is the first domain and a remote sensing image whose corresponding domain is the second domain, and generates a trained model that infers the displacement amount corresponding to the input; A learning device comprising: An alignment system comprising: the first learning image is the first simulated image after transformation when the first simulated image is transformed, and is the first simulated image when the first simulated image is not transformed; The second training image is the second simulated image after deformation when the second simulated image is deformed, and is the second simulated image when the second simulated image is not deformed.

2. generating a simulated remote sensing image, which indicates the target range and corresponds to the first domain, as a first simulated image based on first ground surface information which indicates the ground surface in the target range; a simulated image generating unit that generates, as a second simulated image, a remote sensing image that indicates the target range and corresponds to a second domain different from the first domain, based on second ground surface information that indicates the ground surface in the target range; an input image generation device comprising: an image deformation unit that, when a positional deviation amount indicating an amount of positional deviation to be set between the first simulated image and the second simulated image is generated, deforms at least one of the first simulated image and the second simulated image based on the positional deviation amount; A learning device comprising: An alignment system comprising: The image deformation unit is an alignment system that further deforms at least one of the first simulated image and the second simulated image so that, assuming that the ground surface in the target range changes between two periods, one of the first simulated image and the second simulated image becomes a remote sensing image showing the ground surface before the change, and the other of the first simulated image and the second simulated image becomes a remote sensing image showing the ground surface after the change.

3. generating a simulated remote sensing image, which indicates the target range and corresponds to the first domain, as a first simulated image based on first ground surface information which indicates the ground surface in the target range; a simulated image generating unit that generates, as a second simulated image, a remote sensing image that indicates the target range and corresponds to a second domain different from the first domain, based on second ground surface information that indicates the ground surface in the target range; an input image generation device comprising: an image deformation unit that, when a positional deviation amount indicating an amount of positional deviation to be set between the first simulated image and the second simulated image is generated, deforms at least one of the first simulated image and the second simulated image based on the positional deviation amount; A learning device comprising: An alignment system comprising: The image deformation unit is an alignment system that, when it is assumed that an external disturbance occurs when capturing a remote sensing image, deforms at least one of the first simulated image and the second simulated image, thereby further reflecting the influence of the external disturbance on at least one of the first simulated image and the second simulated image.

4. generating a simulated remote sensing image, which indicates the target range and corresponds to the first domain, as a first simulated image based on first ground surface information which indicates the ground surface in the target range; a simulated image generating unit that generates, as a second simulated image, a remote sensing image that indicates the target range and corresponds to a second domain different from the first domain, based on second ground surface information that indicates the ground surface in the target range; an input image generation device comprising: an image deformation unit that, when a positional deviation amount indicating an amount of positional deviation to be set between the first simulated image and the second simulated image is generated, deforms at least one of the first simulated image and the second simulated image based on the positional deviation amount; A learning device comprising: An alignment system comprising: at least one of the first ground surface information and the second ground surface information is model information indicating a model of the ground surface in the target range, The simulated image generation unit When the first earth's surface information is the model information, the first simulated image is generated based on information indicating a first imaging condition, which is an imaging condition of a remote sensing image corresponding to the first domain; When the second earth surface information is the model information, an alignment system further generates the second simulated image based on information indicating a second imaging condition, which is an imaging condition of a remote sensing image corresponding to the second domain.

5. The learning device further a model generation unit that uses learning data including a first learning image, a second learning image, and the positional displacement amount to learn a relationship between the first learning image and the second learning image and the positional displacement amount, thereby generating a trained model that infers a positional displacement amount corresponding to the first inference image and the second inference image from the first inference image, which is a remote sensing image whose corresponding domain is the first domain, and the second inference image, which is a remote sensing image whose corresponding domain is the second domain. Equipped with When the first simulated image is transformed, the first learning image is the transformed first simulated image, When the first simulated image is not transformed, the first training image is the first simulated image; When the second simulated image is transformed, the second learning image is the second simulated image after the transformation, The registration system of claim 2 , wherein the second training image is the second simulated image when the second simulated image is not deformed.

6. The input image generation device further comprises: a positional deviation amount generating unit for generating the positional deviation amount; The alignment system according to claim 1 , comprising:

7. The learning device further a positional deviation amount generating unit for generating the positional deviation amount; The alignment system according to claim 1 , comprising:

8. The alignment system described in claim 1, 3, or 4, wherein the image deformation unit further deforms at least one of the first simulated image and the second simulated image so that, assuming that the ground surface in the target range changes between two periods, one of the first simulated image and the second simulated image becomes a remote sensing image showing the ground surface before the change, and the other of the first simulated image and the second simulated image becomes a remote sensing image showing the ground surface after the change.

9. The alignment system of claim 1, 2, or 4, wherein the image deformation unit, assuming that an external disturbance occurs when capturing a remote sensing image, deforms at least one of the first simulated image and the second simulated image to further reflect the influence of the external disturbance on at least one of the first simulated image and the second simulated image.

10. at least one of the first ground surface information and the second ground surface information is model information indicating a model of the ground surface in the target range, The simulated image generation unit When the first earth's surface information is the model information, the first simulated image is generated based on information indicating a first imaging condition, which is an imaging condition of a remote sensing image corresponding to the first domain; An alignment system described in any one of claims 1 to 3, wherein when the second earth's surface information is the model information, the second simulated image is further generated based on information indicating a second imaging condition, which is the imaging condition of a remote sensing image corresponding to the second domain.

11. The alignment system according to claim 1 , wherein the simulated image generation unit generates the first simulated image and the second simulated image using Image Translation AI (Artificial Intelligence).

12. the first ground surface information is a target optical image that is an optical image showing the target range, the first domain is a domain of an optical image; The alignment system according to claim 1 , wherein the simulated image generator generates the first simulated image from the target optical image by adjusting each parameter of the target optical image.

13. the second ground surface information is model information indicating a model of the ground surface in the target range, the second domain is a domain of synthetic aperture radar images; 5. The registration system according to claim 1, wherein the simulated image generation unit generates the second simulated image using an RCS (Radar Cross-Section) simulator based on the model information.

14. the first earth surface information is a remote sensing image corresponding to the second domain; The alignment system described in any one of claims 1 to 4, wherein the simulated image generation unit generates the first simulated image using a trained model that has learned the relationship between a remote sensing image corresponding to the second domain and a remote sensing image corresponding to the first domain, and that uses the first earth surface information as input and infers the first earth surface information and the remote sensing image corresponding to the first domain.

15. the second earth surface information is a remote sensing image corresponding to the first domain; The alignment system described in any one of claims 1 to 4, wherein the simulated image generation unit generates the second simulated image using a trained model that has learned the relationship between a remote sensing image corresponding to the first domain and a remote sensing image corresponding to the second domain, and that uses the second earth surface information as input and infers the second earth surface information and the remote sensing image corresponding to the second domain.

16. the first domain is a domain of an optical image; The alignment system according to claim 1 , wherein the second domain is a domain of a synthetic aperture radar image.

17. an input image generating device which is a computer, based on first ground surface information and second ground surface information indicating the ground surface of a target range, simulates generating a first simulated image which is a remote sensing image corresponding to the target range and a first domain, and a second simulated image which is a remote sensing image corresponding to the target range and a second domain different from the first domain; The learning device, which is a computer, When a positional deviation amount indicating an amount of positional deviation set between the first simulated image and the second simulated image is generated, at least one of the first simulated image and the second simulated image is deformed based on the positional deviation amount; A registration method for generating a trained model that uses training data including a first training image, a second training image, and the displacement amount to learn a relationship between the first training image, the second training image, and the displacement amount, and that receives as input a remote sensing image whose corresponding domain is the first domain and a remote sensing image whose corresponding domain is the second domain and generates a trained model that infers a displacement amount corresponding to the input, the first learning image is the first simulated image after transformation when the first simulated image is transformed, and is the first simulated image when the first simulated image is not transformed; A registration method in which the second training image is the second simulated image after deformation when the second simulated image is deformed, and is the second simulated image when the second simulated image is not deformed.

18. An input image generating device that is a computer, generating a simulated remote sensing image, which indicates the target range and corresponds to the first domain, as a first simulated image based on first ground surface information which indicates the ground surface in the target range; generating, as a second simulated image, a simulated remote sensing image that indicates the target range and corresponds to a second domain different from the first domain, based on second ground surface information that indicates the ground surface in the target range; a learning device that is a computer, when a positional deviation amount indicating an amount of positional deviation set between the first simulated image and the second simulated image is generated, transforming at least one of the first simulated image and the second simulated image based on the positional deviation amount, The learning device is an alignment method in which, assuming that the ground surface in the target range changes between two periods, the learning device further deforms at least one of the first simulated image and the second simulated image so that one of the first simulated image and the second simulated image becomes a remote sensing image showing the ground surface before the change, and the other of the first simulated image and the second simulated image becomes a remote sensing image showing the ground surface after the change.

19. An input image generating device that is a computer, generating a simulated remote sensing image, which indicates the target range and corresponds to the first domain, as a first simulated image based on first ground surface information which indicates the ground surface in the target range; generating, as a second simulated image, a simulated remote sensing image that indicates the target range and corresponds to a second domain different from the first domain, based on second ground surface information that indicates the ground surface in the target range; a learning device that is a computer, when a positional deviation amount indicating an amount of positional deviation set between the first simulated image and the second simulated image is generated, transforming at least one of the first simulated image and the second simulated image based on the positional deviation amount, The learning device is an alignment method in which, assuming that external disturbances occur when capturing remote sensing images, the learning device deforms at least one of the first simulated image and the second simulated image, thereby further reflecting the effects of the disturbances in at least one of the first simulated image and the second simulated image.

20. An input image generating device that is a computer, generating a simulated remote sensing image, which indicates the target range and corresponds to the first domain, as a first simulated image based on first ground surface information which indicates the ground surface in the target range; generating, as a second simulated image, a simulated remote sensing image that indicates the target range and corresponds to a second domain different from the first domain, based on second ground surface information that indicates the ground surface in the target range; a learning device that is a computer, when a positional deviation amount indicating an amount of positional deviation set between the first simulated image and the second simulated image is generated, transforming at least one of the first simulated image and the second simulated image based on the positional deviation amount, at least one of the first ground surface information and the second ground surface information is model information indicating a model of the ground surface in the target range, The input image generation device When the first earth's surface information is the model information, the first simulated image is generated based on information indicating a first imaging condition, which is an imaging condition of a remote sensing image corresponding to the first domain; A registration method in which, when the second earth's surface information is the model information, the second simulated image is further generated based on information indicating second imaging conditions, which are imaging conditions of a remote sensing image corresponding to the second domain.

21. A simulated image generation process for simulating, based on first ground surface information and second ground surface information showing the ground surface of a target range, a first simulated image which is a remote sensing image corresponding to the target range and a first domain, and a second simulated image which is a remote sensing image corresponding to the target range and a second domain different from the first domain. an input image generating device which is a computer, an image transformation process for transforming at least one of the first simulated image and the second simulated image based on a positional shift amount when the positional shift amount indicating the amount of positional shift set between the first simulated image and the second simulated image is generated; a model generation process for generating a trained model that uses training data including a first training image, a second training image, and the displacement amount to learn a relationship between the first training image, the second training image, and the displacement amount, and that receives as input a remote sensing image whose corresponding domain is the first domain and a remote sensing image whose corresponding domain is the second domain, and infers the displacement amount corresponding to the input; A registration program that causes a learning device that is a computer to execute the above the first learning image is the first simulated image after transformation when the first simulated image is transformed, and is the first simulated image when the first simulated image is not transformed; An alignment program in which the second learning image is the second simulated image after deformation when the second simulated image is deformed, and is the second simulated image when the second simulated image is not deformed.

22. generating a simulated remote sensing image, which indicates the target range and corresponds to the first domain, as a first simulated image based on first ground surface information which indicates the ground surface in the target range; a simulated image generation process for generating, as a second simulated image, a remote sensing image that indicates the target range and corresponds to a second domain different from the first domain, based on second ground surface information that indicates the ground surface in the target range; an input image generating device which is a computer, an image transformation process for transforming at least one of the first simulated image and the second simulated image based on a positional shift amount when the positional shift amount indicating the amount of positional shift set between the first simulated image and the second simulated image is generated; A registration program that causes a learning device that is a computer to execute the above In the image transformation process, when it is assumed that the ground surface in the target range changes between two periods, an alignment program further transforms at least one of the first simulated image and the second simulated image so that one of the first simulated image and the second simulated image becomes a remote sensing image showing the ground surface before the change, and the other of the first simulated image and the second simulated image becomes a remote sensing image showing the ground surface after the change.

23. generating a simulated remote sensing image, which indicates the target range and corresponds to the first domain, as a first simulated image based on first ground surface information which indicates the ground surface in the target range; a simulated image generation process for generating, as a second simulated image, a remote sensing image that indicates the target range and corresponds to a second domain different from the first domain, based on second ground surface information that indicates the ground surface in the target range; an input image generating device which is a computer, an image transformation process for transforming at least one of the first simulated image and the second simulated image based on a positional shift amount when the positional shift amount indicating the amount of positional shift set between the first simulated image and the second simulated image is generated; A registration program that causes a learning device that is a computer to execute the above In the image deformation process, assuming that an external disturbance occurs when capturing a remote sensing image, at least one of the first simulated image and the second simulated image is deformed, thereby further reflecting the influence of the external disturbance on at least one of the first simulated image and the second simulated image.

24. generating a simulated remote sensing image, which indicates the target range and corresponds to the first domain, as a first simulated image based on first ground surface information which indicates the ground surface in the target range; a simulated image generation process for generating, as a second simulated image, a remote sensing image that indicates the target range and corresponds to a second domain different from the first domain, based on second ground surface information that indicates the ground surface in the target range; an input image generating device which is a computer, an image transformation process for transforming at least one of the first simulated image and the second simulated image based on a positional shift amount when the positional shift amount indicating the amount of positional shift set between the first simulated image and the second simulated image is generated; A registration program that causes a learning device that is a computer to execute the above at least one of the first ground surface information and the second ground surface information is model information indicating a model of the ground surface in the target range, In the simulated image generation process, When the first earth's surface information is the model information, the first simulated image is generated based on information indicating a first imaging condition, which is an imaging condition of a remote sensing image corresponding to the first domain; and an alignment program that, when the second earth's surface information is the model information, further generates the second simulated image based on information indicating second imaging conditions, which are imaging conditions of a remote sensing image corresponding to the second domain.

Images