Information processing system and information processing method

Abstract

To make observations more efficient by utilizing 3D data. [Solution] The information processing system includes a first generation unit, an acquisition unit, an extraction unit, a matching unit, and an assignment unit. The first generation unit generates an RGB image corresponding to a predetermined synthetic aperture radar image based on a predetermined synthetic aperture radar image and a predetermined machine learning model. The acquisition unit acquires a predetermined optical image corresponding to the predetermined synthetic aperture radar image, the optical image having first position information assigned to it. The extraction unit extracts feature points in a first area indicated by the optical image and a second area indicated by the RGB image. The matching unit matches the feature points between the first and second areas. The assignment unit assigns second position information corresponding to the first position information to the RGB image based on the matching result.

Description

[Technical Field]

[0001] The present invention relates to an information processing system and an information processing method. [Background technology]

[0002] In recent years, observations using three-dimensional data have been conducted.

[0003] For example, Patent Document 1 proposes a method for accurately detecting anti-aircraft signs from captured images using images of the anti-aircraft signs in order to measure buildings, soil volume, and the like. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2022-133328 Summary of the Invention [Means for solving the problem]

[0005] The information processing system according to the present application includes a first generation unit that generates an RGB image corresponding to a predetermined synthetic aperture radar image based on the synthetic aperture radar image and a predetermined machine learning model; an acquisition unit that acquires a predetermined optical image corresponding to the predetermined synthetic aperture radar image, the optical image having first position information assigned to it; an extraction unit that extracts feature points in a first area indicated by the optical image and a second area indicated by the RGB image; a matching unit that matches the feature points between the first area and the second area; and an assignment unit that assigns second position information corresponding to the first position information to the RGB image based on a result of the matching. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 shows an overview of earth surface observation using SAR interferometry. [Figure 2]FIG. 2 is a diagram showing an overall view of the absolute coordinate assignment process. [Figure 3] FIG. 3 is a diagram showing an overall view of the 3D conversion process. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of a learning method according to the embodiment. [Figure 6] FIG. 6 is a diagram showing a specific example of feature point matching according to the embodiment. [Figure 7] FIG. 7 is a flowchart showing the procedure of the pre-processing PR1. [Figure 8] FIG. 8 is a flowchart showing the procedure of the pre-processing PR2. [Figure 9] FIG. 9 is a diagram showing a specific example of adding location information. [Figure 10] FIG. 10 is a flowchart showing the procedure of the colorization process. [Figure 11] FIG. 11 is a flowchart showing the procedure for feature point matching. [Figure 12] FIG. 12 is a flowchart (1) showing the procedure of the 3D conversion process. [Figure 13] FIG. 13 is a flowchart (2) showing the procedure of the 3D conversion process. [Figure 14] FIG. 14 is a flowchart showing the procedure for calculating the volume of soil and sand. [Figure 15] FIG. 15 is a flowchart showing the procedure of the control process for removing earth and sand. [Figure 16] FIG. 16 is a diagram showing an overall view of the modified example. [Figure 17] FIG. 17 is a flowchart showing a procedure for feature point matching according to a modified example. [Figure 18] FIG. 18 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0008] One or more embodiments (including examples, modifications, and application examples) described below can be implemented independently. However, at least a portion of the embodiments described below may be implemented in appropriate combination with at least a portion of another embodiment. These embodiments may include novel features that are different from each other. Therefore, these embodiments may contribute to solving different purposes or problems and may produce different effects from each other.

[0009] In the following embodiments, optical images include optical satellite images taken by an optical satellite or aerial images taken from the sky by an aircraft such as a drone. Furthermore, the events targeted by the proposed technology of the present invention (hereinafter sometimes simply referred to as the "proposed technology") will be described as natural disasters caused by critical natural phenomena, but the proposed technology of the present invention can be applied to various events regardless of natural disasters.

[0010] Natural disasters include storms, heavy rain, heavy snowfall, floods, high tides, earthquakes, tsunamis, and volcanic eruptions. In the following embodiment, the proposed technology will be described with a focus on landslides.

[0011] <Embodiment> 1. Introduction (About SAR) Observations using synthetic aperture radar (SAR) are well known. SAR is not a passive sensor that uses electromagnetic energy emitted from the sun, but rather an active sensor that irradiates microwaves from the radar antenna itself toward the object being observed and receives the electromagnetic energy scattered by the object with an antenna, making it possible to make observations both day and night. Furthermore, SAR irradiates microwaves with wavelengths longer than the size of particles in the atmosphere such as clouds and rain, so it is less affected by cloud cover. As such, SAR has the advantage of being able to perform observations in all weather conditions, regardless of the time of day or weather conditions.

[0012] Furthermore, because SAR can capture objects with a spatial resolution that is not possible with other observation methods, it is also used to observe landslides and other sediment disasters, as well as deformation of the earth's surface due to land subsidence, etc. For example, a satellite-mounted SAR can emit microwaves in a direction perpendicular to the satellite's direction of travel and obliquely downward, observe the backscattered waves from the earth's surface, and analyze the observed data to obtain SAR images with high spatial resolution.

[0013] SAR images store information on scattering intensity and phase for each pixel. Scattering intensity includes information on the slope, roughness, and dielectric constant of the Earth's surface, while phase includes information on the distance from the satellite to the pixel and the scattering pattern. While SAR images are intensity images that also include phase information, it is difficult to observe changes in the Earth's surface using just one image.

[0014] To address this issue, a technique called SAR interferometry (interferometric SAR) is known. SAR interferometry is a technique for observing the shape and changes of the Earth's surface at an observation location by observing the same location multiple times using a satellite-mounted SAR. FIG. 1 shows an overview of Earth's surface observation using SAR interferometry. FIG. 1 shows a scene in which two SAR observations are carried out on the same location on the Earth's surface using a SAR satellite 10, which is an artificial satellite equipped with a SAR.

[0015] In this way, by conducting two SAR observations to prepare two SAR images (phase images), and then interferometric analysis (also called SAR analysis or interferometry), which involves interferometrically detecting the difference between the two images, it is possible to utilize information on the slight distance difference Δd and detect deformation on the Earth's surface. Furthermore, the results of interferometric analysis are generally expressed as an interferogram, which displays deformation as rainbow-colored fringes. As shown in the example in Figure 1, the interferogram is created by interfering the SAR image from the first observation with the SAR image from the second observation, resulting in an interferogram with interference fringes such as topographic fringes and orbital fringes. For this reason, SAR interferometry can visualize crustal deformation associated with an earthquake by performing interferometric analysis using two observational data acquired before and after the earthquake, for example.

[0016] SAR images have several processing levels, but the data immediately after imaging, which serves as the base product for subsequent processing, is also called SLC (Single Look Complex) data. SLC data is the highest resolution SAR image, and contains not only amplitude information that can be recognized as an image, but also phase information.

[0017] For example, in interferometry analysis, a differential interferogram is generated using the SLC data obtained in the first observation and the SLC data obtained in the second observation, using the following procedure: Specifically, the SLC data obtained in the first observation is aligned with the SLC data obtained in the second observation (Step 1), an interferometric process is performed to generate an interferogram from the two SLC data (Step 2), and a differential interferogram is generated by removing the interference fringes from the interferogram (Step 3).

[0018] Here, in step 1, alignment, the difference in corresponding pixel coordinates between the SLC data obtained in the first observation and the SLC data obtained in the second observation is measured at the sub-pixel level, and a conversion coefficient is calculated to calculate the pixel position of the SLC data obtained in the second observation that corresponds to the pixel in the SLC data obtained in the first observation. The two SLC data are then aligned based on this conversion coefficient. This type of alignment has the problem of requiring time-consuming work such as manually overlaying landmarks, etc. Below, we will explain in detail the background and challenges that led to the proposed technology.

[0019] (Task 1) Rapid responses are required when disasters occur, and by comparing 3D models (3D point cloud data) acquired by laser scanners, it is possible to distinguish between damaged and normal areas, for example, and this has been found to be useful in local government response and rescue efforts. However, there are limited means of generating high-precision 3D models of the entire country (for example, all of Japan), and while there is a need to quickly generate high-precision 3D models over a wide area, current methods are time-consuming and costly, which presents a problem.

[0020] Therefore, the inventors of the present invention have focused on the possibility that if SAR, which can capture high-resolution images in all weather conditions regardless of the time of day or weather conditions, is used, it may be possible to generate highly accurate 3D models using SAR images as source images. Furthermore, SAR missions that can observe at higher frequencies are now being realized, and multiple SAR satellites 10 are being operated simultaneously. Therefore, it is possible to observe the same location multiple times a day using multiple SAR satellites 10, and SAR is considered useful in terms of accelerating disaster response.

[0021] On the other hand, when generating a 3D model from SAR images, the SAR images are aligned with each other, but as mentioned above, this alignment requires manual work, which is time-consuming and costly. This is because SAR images do not have absolute coordinates, making it difficult to identify positions within the SAR image. For example, with SAR images, relative coordinate changes can be monitored from phase changes on different time axes, but the SAR image itself does not have absolute coordinates, so the operator of the SAR satellite 10 must perform the time-consuming task of overlaying SAR images to represent coordinates based on the positions of landmarks and other objects contained in the SAR image.

[0022] In this regard, the inventors of the present invention have discovered that by combining RTK (Real Time Kinematic) positioning technology, which can determine location information with a high accuracy of within a few centimeters, with SAR observation technology, it is possible to assign absolute coordinates to SAR images.

[0023] (Task 2) From here, we will explain the background and issues that led to the proposed technology from a different perspective than the above.For example, one possible method for assigning absolute coordinates to SAR images is to extract feature points of the shooting area from optical images, which have absolute coordinates based on the position coordinates of the reference station used for RTK positioning, and SAR images, which do not have absolute coordinates, and then assign absolute coordinates to the SAR images by feature point matching using the extracted feature points.

[0024] However, there are cases where feature point matching between optical images and SAR images cannot be performed accurately. Specifically, the principle of optical image capture is to measure scattered light from sunlight using remote sensing technology and analyze the properties of objects measured at long distances. For this reason, optical images are generated as full-color RGB images.

[0025] On the other hand, the imaging principle of SAR images is to analyze the properties of objects measured at long distances by observing the reflected waves when radio waves are irradiated onto the object. For this reason, SAR images are generated as grayscale black and white images.

[0026] As such, optical images and SAR images may not be able to match feature points properly due to the different imaging principles. It is possible to generate a three-dimensional model using only optical images, without using SAR images. However, optical satellites have various drawbacks. These drawbacks will be explained using optical satellite images acquired by an optical satellite as an example of an optical satellite.

[0027] For example, optical satellites have been launched far fewer times than SAR satellites,10 and the rate at which optical satellite images of the same location can be acquired is approximately one image per 14 days, which is slower than the rate at which SAR images of the same location can be acquired (approximately one image per 10 minutes). For this reason, the use of optical satellite images lacks real-time capabilities, making them impractical in situations where immediate action is required, such as in the event of a disaster. Optical satellite images also have the disadvantage of being susceptible to the effects of time of day and weather conditions, as they cannot be observed in areas covered by clouds or at night.

[0028] For this reason, the inventors of the present invention have focused on the use of SAR images. As described above, SAR satellites 10 can observe the same location at a rate of about one image per 10 minutes. Therefore, the use of SAR images has the advantage of being superior in real-time performance and enabling observation independent of time of day and weather conditions, thereby overcoming the drawbacks of using optical satellite images. However, SAR images have the problem that they are not suitable for feature point matching to assign absolute coordinates.

[0029] (Summary of proposed technology) Based on the above, the proposed technology first colorizes the grayscale SAR image to generate a pseudo-optical image, which is a color image (i.e., an RGB image) that approximates the color of the optical image. In this state, the proposed technology extracts feature points of the shooting area from both the optical image, which has absolute coordinates, and the pseudo-optical image, which does not have absolute coordinates, and then performs feature point matching using the extracted feature points to assign absolute coordinates to the pseudo-optical image. As a result, the proposed technology makes it possible to identify the position relative to the pseudo-optical image, and a three-dimensional model is generated from the pseudo-optical image to which absolute coordinates have been assigned.

[0030] This proposed technology will enable the efficient and highly accurate observation of the area by utilizing three-dimensional data.

[0031] Furthermore, the proposed technology is a new technology that improves the accuracy of remote sensing data, including SAR images, by correcting the absolute coordinates of optical images. It overcomes the problem that conventional SAR images do not have absolute coordinates, and enables highly accurate three-dimensional analysis and the creation of GIS data, making it suitable for a wide range of industrial applications. Therefore, the proposed technology can serve as an innovative technological foundation for the telecommunications industry, for example, and contribute to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and resilience."

[0032] [2. Overview of the proposed technology] From here, an overview of the proposed technology of the present invention will be explained using Figures 2 and 3. Figure 2 explains the overall picture of the absolute coordinate assignment process in which absolute coordinates are assigned to SAR images. Figure 3 explains the overall picture of the 3D processing in which a three-dimensional model is generated from SAR images to which absolute coordinates have been assigned.

[0033] (Absolute coordinate assignment processing) Fig. 2 is a diagram showing an overall view of the absolute coordinate assignment process. Fig. 2 shows an information processing system Sy in which information processing according to the proposed technology is realized. The information processing system Sy may include a SAR observation system 1 that performs SAR observation using a SAR mounted on a SAR satellite 10, and an RTK positioning system 2. The mechanism of SAR observation in the SAR observation system 1 has been described above, so a description thereof will be omitted.

[0034] The RTK positioning system 2 may include a receiving device 20, a reference station 30, a Global Navigation Satellite System (GNSS) satellite 40, and a positioning core system 50.

[0035] The receiving device 20 may be a portable information processing terminal and may be installed, for example, anywhere a user desires to know location information. The receiving device 20 may receive GNSS signals from GNSS satellites 40. That is, the receiving device 20 may be equipped with a GNSS module (positioning module) and antenna corresponding to a GNSS receiver in RTK. The receiving device 20 may also be equipped with a communication module for communicating with the centralized core system 50.

[0036] The reference station 30 may function as a reference station in the RTK calculation. That is, the reference station 30 may have known coordinates (known coordinates) that indicate its own position. Furthermore, if there are multiple reference stations 30, each of the multiple reference stations 30 may have known coordinates. The reference station 30 may receive GNSS signals from GNSS satellites 40.

[0037] In RTK positioning, first, the receiving device 20 and multiple reference stations 30, whose precise position information (absolute coordinates) is known, receive GNSS signals from GNSS satellites 40 and each acquires its own position information based on the received GNSS signals. Next, the receiving device 20, which is attempting to determine its position, transmits its own rough position information (hereinafter referred to as approximate position information) obtained based on the GNSS signals to the centralized core system 50. Then, based on the transmitted approximate position information, the centralized core system 50 acquires in real time the position information of each reference station 30 obtained based on the GNSS signals from multiple reference stations 30 located near the receiving device 20.

[0038] Next, the positioning core system 50 generates correction information by calculating the difference between the known, precise position information of the reference station 30 (for example, stored in advance in the positioning core system 50) and the position information obtained based on the GNSS signal acquired from the reference station 30. The correction information is used to perform real-time corrections to the approximate position information of the receiving device 20 obtained based on the GNSS signal. That is, in RTK positioning, the approximate position information of the receiving device 20 obtained based on the GNSS signal is corrected using correction information generated from the position information (position information obtained based on the GNSS signal, precise position information) of the reference station 30 installed in a reception environment estimated to be similar to the reception environment of the GNSS signal of the receiving device 20 whose position is to be determined.

[0039] According to such RTK positioning, by performing the corrections described above, it is possible to correct errors caused by ionospheric disturbances, multipath, clock misalignment, etc. that occur in GNSS signals, and therefore errors contained in the position information can be reduced to about a few centimeters. As a result, according to RTK positioning, it is easy to obtain position information of the receiving device 20 with little error.

[0040] In the RTK positioning that can be used in this embodiment, it is preferable that the plurality of reference stations 30 are installed so that the distance between the reference stations 30 is within 10 kilometers.

[0041] Furthermore, in RTK positioning, the core system 50 or the receiving device 20 uses the correction information generated by the core system 50 to correct the approximate position information obtained based on the GNSS signal, thereby obtaining position information of the receiving device 20 with little error. More specifically, RTK positioning can be mainly divided into two methods: a device RTK positioning method in which the correction is performed on the receiving device 20 side, and a server RTK positioning method in which the correction is performed on the core system 50 side.

[0042] In the absolute coordinate assignment process implemented by the information processing system Sy, known accurate position information for each of the multiple reference stations 30 is used.

[0043] Here, the reference station image BG including the reference station 30 is an optical satellite image including the reference station 30 taken by an optical satellite 60, as shown in FIG. 2, and belongs to the category of full-color optical images. The reference station image BG also has position information of the reference station 30. As described above, this position information of the reference station 30 is accurate position information determined in advance for the reference station 30, and is absolute position information (absolute coordinates) when identifying the position of the first area AR (hereinafter referred to as "area AR1"), which is the photographing area AR indicated by the reference station image BG. The number of reference stations 30 included in the reference station image BG is not limited.

[0044] In this state, in the absolute coordinate assignment process, first position information based on the accurate position information of the reference station 30 is assigned to the reference station image BG. Specifically, in the absolute coordinate assignment process, position information based on the accurate position information of the reference station 30, and position information of objects in the area AR1 around the reference station 30 is calculated as the first position information. Then, the first position information is assigned to the image portion of the object in the reference station image BG. The first position information can be said to be absolute position information (absolute coordinates) based on the accurate position information of the reference station 30.

[0045] Furthermore, the SAR satellite 10 captures images while orbiting along a specific orbit. Therefore, sequential SAR observations according to the orbit allow an SAR image SG of the entire country (for example, the entire country of Japan) to be obtained, just like the reference station image BG. The SAR image SG itself does not have absolute coordinates, but absolute coordinates are assigned by an absolute coordinate assignment process.

[0046] In the absolute coordinate assignment process, absolute coordinates are assigned to the SAR image SG, and feature point matching with the reference station image BG is performed. However, as described above, the reference station image BG, which is an optical image, and the SAR image SG, which is a grayscale image, are captured using different imaging principles, and therefore, appropriate feature point matching may not be possible.

[0047] Therefore, in the absolute coordinate assignment process, the SAR image SG is colorized to generate a pseudo-optical image CL_SG, which is a full-color image (i.e., an RGB image) that approximates the color tone of the optical image. Specifically, in the absolute coordinate assignment process, a pseudo-optical image CL_SG is generated by colorizing the SAR image SG based on the model M1, which is an AI model trained to pseudo-reproduce an optical image corresponding to the input SAR image SG, and the SAR image SG to which absolute coordinates are to be assigned. Hereinafter, the SAR image SG to which absolute coordinates are to be assigned (a SAR image SG that is not colorized and does not have absolute coordinates) will be referred to as the "original SAR image SG."

[0048] FIG. 2 shows an example in which a pseudo optical image CL_SG is generated in accordance with a second area AR (hereinafter referred to as "area AR2"), which is the imaging area AR indicated by the original SAR image SG, with the area AR2 as the imaging area AR.

[0049] At this point, a base station image BG having absolute coordinates and a pseudo optical image CL_SG without absolute coordinates have been obtained. Therefore, in the absolute coordinate assignment process, feature points F in area AR1 indicated by the base station image BG and area AR2 indicated by the pseudo optical image CL_SG without absolute coordinates are extracted. For example, in the absolute coordinate assignment process, a first feature point F (hereinafter referred to as "feature point F1") that is feature point F in area AR1 is extracted, and a second feature point F (hereinafter referred to as "feature point F2") that is feature point F in area AR2 is extracted.

[0050] The algorithm for extracting feature points may be a model M2, which is an AI model based on a predetermined deep learning. The model M2 may be, for example, SIFT (Scale-Invariant Feature Transform), AKAZE (Accelerated-KAZE), ORB (Oriented FAST and Rotated BRIEF), etc.

[0051] Next, in the absolute coordinate assignment process, a matching process is executed to match the common feature points F between the feature points F1 and F2.

[0052] Then, in the absolute coordinate assignment process, second position information corresponding to the first position information is assigned to the pseudo optical image CL_SG, which does not have absolute coordinates, based on the results of the matching process. For example, in the absolute coordinate assignment process, information transfer may be performed to transfer the second position information to all feature points F2 included in a feature point pair matched between feature point F1 and feature point F2, and the feature point F1 indicating the object for which the first position information was calculated and the matched feature point F2. Note that in the absolute coordinate assignment process, the second position information may also be transferred on a pixel-by-pixel basis using the second position information assigned to all feature points F2. Similar to the first position information, the second position information can be considered absolute position information (absolute coordinates) based on the accurate position information of the reference station 30.

[0053] FIG. 2 shows an example in which a pseudo optical image CL_SG′ is obtained as a pseudo optical image CL_SG having absolute coordinates by the absolute coordinate assignment process.

[0054] (3D processing) FIG. 3 is a diagram showing an overall view of the 3D rendering process. The 3D rendering process may also be executed in the information processing system Sy. As shown in FIG. 3, the 3D rendering process may be executed using an original SAR image SG and a pseudo optical image CL_SG' obtained by colorizing the original SAR image SG and assigning absolute coordinates. The 3D rendering process here requires processing to determine the unevenness of the ground surface (specifically, processing to detect the area of ​​an object OB included in the image and to determine the height of the object in the detected area), and the obtained information is assigned to the pseudo optical image CL_SG' obtained in the absolute coordinate assignment process.

[0055] This will be explained more specifically using the example of Figure 3. In the 3D conversion process, as shown in Figure 3, model M3, which is an AI model trained to detect the area of ​​a specific object OB (for example, a structure such as a house or building) in pixel units from the SAR image, and model M4, which is an AI model trained to estimate the height of the detected area from the SAR image, may be used.

[0056] Model M3 may be a so-called semantic segmentation model, which can be called a "building extraction AI." Model M4 may be called a "height inference AI." Both Model M3 and Model M4 are machine learning models newly trained and generated to realize the proposed technology, and may be, for example, a convolutional neural network (CNN).

[0057] According to the example of Figure 3, in the 3D conversion process, based on the output result output by the model M3 for the input original SAR image SG, the area of ​​the object OB in the area AR2 indicated by the original SAR image SG is detected, and object area information OB_DA2, which is information on the detected area, is obtained.

[0058] According to the example of Figure 3, in the 3D conversion process, the height of the object OB in the area AR2 indicated by the original SAR image SG is estimated based on the output result output by the model M4 for the input original SAR image SG, and object height information OB_DA3, which is information on the estimated height, is obtained.

[0059] Furthermore, in the 3D conversion process, object information OB_DA including object region information OB_DA2 and object height information OB_DA3 is generated, and the object information OB is associated with the original SAR image SG to synthesize a SAR image OB_SG having the object information OB_DA. As shown in Fig. 3, the SAR image OB_SG has the object information OB_DA but does not have absolute coordinates.

[0060] At this point, a SAR image OB_SG having object information OB_DA and a pseudo optical image CL_SG' having absolute coordinates have been obtained. Therefore, in the 3D conversion process, feature points F in the area AR2 indicated by the SAR image OB_SG and the area AR2 indicated by the pseudo optical image CL_SG' are extracted. For example, in the 3D conversion process, a third feature point (hereinafter referred to as "feature point F3") that is a feature point on the SAR image OB_SG side in the area AR2, and a fourth feature point (hereinafter referred to as "feature point F4") that is a feature point on the pseudo optical image CL_SG' side in the area AR2 are extracted. A model M2 may be used as the algorithm for extracting feature points.

[0061] Next, in the 3D conversion process, a matching process is executed to match the common feature points F between the feature points F3 and F4.

[0062] Then, in the 3D conversion process, object information OB_DA is assigned to the pseudo optical image CL_SG' having absolute coordinates based on the result of the matching process. For example, in the 3D conversion process, information transfer may be performed to transfer the object information OB_DA to all feature points F4 included in the pair based on the feature points pair matched between the feature points F3 and F4 and the object information OB_DA associated with the image portion of the paired feature point F3.

[0063] FIG. 3 shows an example in which a pseudo optical image CL_SG'' is obtained by the 3D processing as a pseudo optical image CL_SG that further has object information OB_DA in addition to absolute coordinates.

[0064] In this state, in the 3D conversion process, a three-dimensional model MD is generated in which the object OB, which is represented in a plane in the pseudo-optical image CL_SG'', is represented in a three-dimensional manner based on the object information OB_DA assigned to the pseudo-optical image CL_SG''.

[0065] In addition, in the 3D conversion process, the final three-dimensional model MD' may be generated by processing the three-dimensional model MD to make it closer to the real color according to the RGB elements contained in the pseudo optical image CL_SG''.

[0066] The information processing according to the embodiment described with reference to FIGS. 2 and 3 may be performed by an information processing device 100, which will be described later. Therefore, the information processing device 100 can improve the efficiency of observations using three-dimensional data. For example, the information processing device 100 is installed in an information processing system Sy that includes a SAR observation system 1 and an RTK positioning system 2. This allows for information processing that combines RTK positioning technology and SAR observation technology, enabling the rapid generation of highly accurate three-dimensional models over a wide area without incurring time or cost. As a result, the information processing device 100 can support rapid responses in the event of a disaster. For example, in the event of a landslide caused by an earthquake, the information processing device 100 can estimate the type and number of equipment required to remove the sediment based on the volume of the sediment and notify the local government of the estimated results.

[0067] 3. Configuration of Information Processing Device An information processing device 100 according to an embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of the information processing device 100 according to an embodiment. As shown in Fig. 4, the information processing device 100 may include a communication unit 110, a storage unit 120, and a control unit 130. Note that the information processing device 100 may be a server device realized in the cloud.

[0068] (Communication unit 110) The communication unit 110 is realized by, for example, a network interface card (NIC). For example, the communication unit 110 transmits and receives information to and from the SAR observation system 1 and the RTK positioning system 2. The communication unit 110 may also transmit and receive information to and from a terminal device DV (not shown) of a user U. The user U here may be, for example, an employee of a local government in the area where the disaster occurred.

[0069] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 may store, for example, data and programs related to the information processing according to the embodiment. The storage unit 120 may also store data necessary for the information processing according to the embodiment and data obtained by the information processing according to the embodiment.

[0070] (control unit 130) The control unit 130 is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like executing various programs (for example, the information processing program according to the embodiment) stored in a storage device inside the information processing device 100 using RAM as a work area. The control unit 130 is also realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0071] As shown in Fig. 3, the control unit 130 has an acquisition unit 131, a learning unit 132, a first generation unit 133, an extraction unit 134, a matching unit 135, an assignment unit 136, a second generation unit 137, an estimation unit 138, and a notification unit 139, and realizes or executes the functions and actions of information processing described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Fig. 4, and may be other configurations as long as they perform the information processing described below. Furthermore, the connection relationship between the processing units included in the control unit 130 is not limited to the connection relationship shown in Fig. 4, and may be other connection relationships.

[0072] (Acquisition part 131) The acquisition unit 131 acquires information necessary for information processing according to the embodiment. For example, the acquisition unit 131 acquires optical images and SAR images.

[0073] (Learning Section 132) The learning unit 132 performs learning of a machine learning model for generating image data. For example, the learning unit 132 may use a combination of a sample SAR image and a sample optical image as learning data, and have the model M learn the relationship between the two images, thereby generating an AI model (model M1) that simulates an optical image corresponding to an input SAR image. The learning unit 132 may use a diffusion model as the model M. The learning unit 132 may also perform learning of a model M3 that is a building extraction AI and a model M4 that is a height inference AI.

[0074] (First generation unit 133) The first generation unit 133 generates an RGB image in which the SAR image has been fully colorized from the grayscale SAR image. Specifically, the first generation unit 133 inputs the original SAR image (an example of a predetermined SAR image) into an AI model (model M1) that has been trained to pseudo-reproduce an optical image corresponding to the input SAR image, using a combination of a sample SAR image and a sample optical image as learning data, as a combination of images having similar shooting areas, to generate a pseudo optical image (RGB image) in which an optical image corresponding to the original SAR image is pseudo-reproduced.

[0075] (Extraction part 134) The extraction unit 134 extracts feature points from the image data. For example, the extraction unit 134 extracts feature points in a first area indicated by the optical image and in a photographing area indicated by the pseudo optical image. The extraction unit 134 also extracts feature points in a second area common to an SAR image having object information for the original SAR image and a pseudo optical image generated from the original SAR image, from each image.

[0076] The extraction unit 134 may use SIFT, AKAZE, ORB, or the like as an algorithm (model M2) for extracting feature points. For example, SIFT is an algorithm for feature point detection and feature amount description, which detects feature points from the difference between a smoothed image convolved using DoG (Differences of Gaussian), which approximates LOG (Laplacian of Gaussian), and describes a 128-dimensional gradient vector obtained from surrounding pixel information as a feature amount. SIFT can robustly describe the feature amount of detected feature points against image rotation, scale changes, lighting changes, etc., and can therefore be used for image matching such as image mosaics, and object recognition and detection.

[0077] (Matching section 135) The matching unit 135 performs feature point matching to match feature points between different image data. For example, the matching unit 135 matches feature points between different image data based on feature points and feature amounts. For example, the matching unit 135 calculates a norm as the feature amount of each feature point, and matches feature points that have the smallest distance between the images as common feature points.

[0078] For example, the matching unit 135 matches feature points between an optical image and a pseudo optical image. Furthermore, the matching unit 135 matches feature points between a plurality of pseudo optical images having different information attached thereto. For example, the matching unit 135 matches feature points between an SAR image having object information for the original SAR image and a pseudo optical image generated from the original SAR image.

[0079] (Given unit 136) The assigning unit 136 assigns information to the image data. For example, the assigning unit 136 assigns first position information, which is absolute position information based on the accurate position information of the reference station 30 and is position information assigned to the reference station image, to the reference station image BS (optical satellite image).

[0080] Furthermore, the assigning unit 136 assigns information to the image data based on the matching result by the matching unit 135. For example, the assigning unit 136 assigns second position information corresponding to the first position information to the pseudo optical image (RGB image) as absolute position information based on the accurate position information of the reference station 30. Furthermore, the assigning unit 136 assigns object information assigned to the SAR image to the pseudo optical image generated from the original SAR image.

[0081] (Second generation unit 137) The second generation unit 137 generates a three-dimensional model having the second position information, which is a three-dimensional model that virtually represents the state of the shooting area shown in the pseudo optical image, based on the original SAR image and the pseudo optical image to which the second position information has been assigned.

[0082] As described above, the assigning unit 136 assigns object information to the pseudo optical image based on the result of matching feature points between an SAR image having object information for the original SAR image and the pseudo optical image generated from the original SAR image. Therefore, the second generating unit 137 generates a three-dimensional model having second position information based on the pseudo optical image to which the object information has been assigned.

[0083] (Estimation part 138) The estimation unit 138 performs various estimations. For example, the estimation unit 138 detects sediment portions based on three-dimensional models generated at different times along a time series and estimates the volume of the detected sediment portions. The estimation unit 138 may also estimate the type and number of equipment required to remove the sediment based on the estimated sediment volume.

[0084] For example, the estimation unit 138 may estimate the type and number of equipment required for removing sediment based on rules. Specifically, if the type and number of equipment required for removing sediment for each volume of sediment are defined as estimation rules, the estimation unit 138 may estimate the type and number of equipment required for removing the sediment by comparing the current volume of sediment with the estimation rules.

[0085] As another example, the estimation unit 138 may estimate the type and number of equipment required for sediment removal based on a machine learning model. In this case, the machine learning model may be trained to output the type and number of equipment required for sediment removal when a sediment volume is input.

[0086] (Regarding notification unit 139) The notification unit 139 notifies the user U of various types of information. For example, when it is determined that an event has occurred, the notification unit 139 may notify the user U of information related to the event that has occurred. For example, when it is determined that a landslide has occurred, the notification unit 139 may notify the user U of alert information that includes at least information indicating the occurrence of the landslide and information about the area where the landslide has occurred.

[0087] In addition, the notification unit 139 may notify the user U of information on the volume of sediment, information on the geographical area (landslide area) newly covered by sediment due to the landslide, and information on the type and number of equipment required to remove the sediment.

[0088] [4. Specific examples of learning methods] A specific example of a learning method for generating an AI model (model M1) that simulates an optical image corresponding to an input SAR image will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of a learning method according to an embodiment. Fig. 5 shows a scene in which model M1 is generated using a combination of a sample SAR image SM_SG and a sample optical image SM_TG as learning data LD.

[0089] According to the example of FIG. 5, the learning unit 132 may generate a data set consisting of multiple pieces of learning data LD, with a pair of an optical image SM_TG and an SAR image SM_SG having similar (or common) imaging areas as one piece of learning data LD.

[0090] The learning unit 132 may then input the data set into, for example, a pre-trained diffusion model, model M, to learn it to estimate RGB elements for each pixel of a grayscale SAR image.

[0091] For example, as shown in FIG. 5, the learning unit 132 may generate a model M1 that outputs a pseudo-optical image CL_SG that is a pseudo-reproduction of an optical image corresponding to the input SAR image SG by having the model M learn the relationship between RGB and grayscale between the optical image SM_TG and the SAR image SM_SG.

[0092] [5. Specific examples of feature point matching] 6A and 6B are diagrams showing a specific example of feature point matching according to the embodiment. Prior to explaining feature point matching, wide-area observation by the SAR satellite 10 will be explained using FIG.

[0093] SAR satellite 10 orbits the Earth, and its orbit includes a path that descends from north to south (southward orbit) and a path that ascends from south to north (northward orbit). When performing feature point matching between reference station image BG and SAR image SG, it is preferable that the observation ranges of the reference station image BG and SAR image SG are the same or that the observation ranges partially overlap.

[0094] FIG. 6(a) shows a scene in which a SAR satellite 10 is performing SAR observation of the ground L while moving in one orbit TR.

[0095] 6(a), the SAR satellite 10 observes ranges R1, R2, R3, and R4 by performing wide-area observation from position P1 on orbit TR. As a result, the information processing device 100 can obtain SAR images SG captured from each of ranges R1, R2, R3, and R4.

[0096] 6(a), the SAR satellite 10 observes ranges R5, R6, R7, and R8 by performing wide-area observation from position P2 on orbit TR. As a result, the information processing device 100 can obtain SAR images SG captured from each of ranges R5, R6, R7, and R8.

[0097] 6(a), the SAR satellite 10 observes ranges R9, R10, R11, and R12 by performing wide-area observation from position P3 on orbit TR. As a result, the information processing device 100 can obtain SAR images SG captured from each of ranges R9, R10, R11, and R12.

[0098] Furthermore, because the SAR satellite 10 observes the same observation range at a predetermined interval while orbiting the Earth along the orbit TR, the information processing device 100 can sequentially acquire SAR images SG of range R1 at a rate of, for example, 6 images per hour (144 images per 24 hours). The same can be said for ranges R2 to R12.

[0099] Next, a specific example of feature point matching according to the embodiment will be described with reference to Fig. 6(b). Fig. 6(b) shows a scene in which feature point matching is performed between a reference station image BG1 and an SAR image SG1, which share a common observation range, range R1. For this reason, the area AR1 indicated by the reference station image BG1 corresponds to range R1. Furthermore, the area AR2 indicated by the SAR image SG1 also corresponds to range R1.

[0100] Furthermore, in the example of FIG. 6(b), the reference station image BG1 includes a reference station 31 (an example of a reference station 30) installed in a structure ST1 (an example of an object OB). Similarly, the SAR image SG1 also includes a reference station 31 installed in the structure ST1. In the example of FIG. 6(b), first position information based on the accurate position information of the reference station 31 is assigned to the location of the structure ST1 in the reference station image BG1. In this example, the first position information is the absolute coordinates of the structure ST1. Therefore, in the example of FIG. 6(b), the reference station image BG1 has absolute coordinates. On the other hand, the SAR image SG1 does not have absolute coordinates in its original state at the time of acquisition. Specifically, the SAR image SG1 does not have absolute coordinates, which are absolute position information of the structure ST1. In addition, the SAR image SG1 is grayscale in its original state.

[0101] Therefore, in the absolute coordinate assignment process for assigning absolute coordinates, prior to feature point matching, the original SAR image SG1 is colorized to generate a pseudo optical image CL_SG, which is a full-color image (i.e., an RGB image) that approximates the color tone of the reference station image BG1, which is an optical satellite image. Specifically, the first generation unit 133 inputs the grayscale original SAR image SG1 to a model M1, and generates a pseudo optical image CL_SG1 that is a pseudo-reproduction of an optical image corresponding to the original SAR image SG1, based on the output result of the model M1.

[0102] At this point, a base station image BG1 having absolute coordinates and a pseudo optical image CL_SG1 having no absolute coordinates have been obtained. Therefore, in the absolute coordinate assignment process, feature points F are extracted in the area AR1 indicated by the base station image BG1 and the area AR2 indicated by the pseudo optical image CL_SG1 having no absolute coordinates. Specifically, the extraction unit 134 uses the model M2 to extract feature point F1, which is the first feature point F in the area AR1, and feature point F2, which is the second feature point F in the area AR2.

[0103] In this state, in the absolute coordinate assignment process, a matching process is executed to match the common feature point F between the feature point F1 and the feature point F2. Specifically, the matching unit 135 calculates norms as feature amounts of the feature point F1 and the feature point F2, and matches the common feature point F between the reference station image BG1 and the pseudo optical image CL_SG1, that is, between the feature point F1 and the feature point F2, with the smallest distance between them.

[0104] Then, in the absolute coordinate assignment process, second position information corresponding to the first position information is assigned to the pseudo optical image CL_SG1, which does not have absolute coordinates, based on the result of the matching process. Specifically, the assignment unit 136 performs information transfer to transfer the second position information to all feature points F2 included in the pair, based on the matched feature point pair between the feature point F1 and the feature point F2, the feature point F2 that matches the feature point F indicating the structure ST1, and the first position information assigned to the location of the structure ST1 in the reference station image BG1. Note that the assignment unit 136 may further assign the second position information on a pixel-by-pixel basis to the pseudo optical image CL_SG1 using the second position information assigned to all feature points F2.

[0105] [6. Processing Procedure] From here, a specific example of the operation of the information processing device 100 will be described. The information processing according to the embodiment may be divided into absolute coordinate assignment processing, 3D processing, and processing as a service utilizing a three-dimensional model. In addition, in the absolute coordinate assignment processing, a pre-processing for assigning absolute coordinates may be executed first. Below, a specific operation procedure of the information processing device 100 in each processing will be described.

[0106] (6-1. Pre-processing (1)) 7 is a flowchart showing the procedure of the pre-processing PR1. The pre-processing PR1 may be a process of using accurate position information of the reference station 30 to assign position information to the object OB.

[0107] As described above, accurate position information is predetermined for each of the reference stations 30 in each location. Therefore, the acquisition unit 131 may acquire the accurate position information of each of the reference stations 30 (step S701). For example, the acquisition unit 131 may acquire the accurate position information of each of the reference stations 30 from the centralized core system 50.

[0108] The acquisition unit 131 may also acquire an installation drawing showing information about the installation location of each reference station 30 (step S702). The reference station 30 is preferably installed in a location where there are no obstacles in the surrounding area that can cause blind spots and where radio waves can be easily received. For this reason, the reference station 30 is often installed on the roof of a building, etc. For this reason, the installation drawing may include, for example, structural information about the structure ST in which the reference station 30 is installed. For example, if the structure ST is a cubic building, the structural information may include information about the length, width, and height of the structure ST. The structure ST may be an example of an object OB located in the vicinity of the reference station 30.

[0109] The assigning unit 136 may calculate position information LI1 of the structure ST based on the accurate position information of the reference station 30 and the installation drawing (step S703). For example, the assigning unit 136 may calculate position information LI1 of the structure ST where the reference station 30 is installed, and link the calculated position information LI1 to the installation drawing as a variable. Specifically, the assigning unit 136 may calculate the position information LI1 of the structure ST as first position information based on the accurate position information of the reference station 30.

[0110] (6-2. Pre-processing (2)) 8 is a flowchart showing the procedure of the pre-processing PR2. The pre-processing PR2 may be a process of adding first position information to an image portion of the object OB in the base station image BG.

[0111] The acquisition unit 131 may acquire a reference station image BS (step S801). There may be multiple reference station images BS depending on the area where the reference station 30 is installed. There may be multiple reference station images BS depending on the observation range of the SAR satellite 10. Furthermore, the reference station image BS may be a full-color optical satellite image of the ground L photographed from above.

[0112] The assigning unit 136 may determine whether or not there is any reference station image BG to which position information has not been assigned (step S802).

[0113] If there is a reference station image BG to which position information has not been assigned (step S802; Yes), the assigning unit 136 may extract one reference station image BG to which position information has not been assigned, and may determine a structure ST included in the extracted reference station image BG (step S803). The structure ST may be an object OB on which the reference station 30 is installed (an object OB located in the vicinity of the reference station 30), and the structure ST can be determined based on an installation drawing corresponding to the reference station 30.

[0114] Next, the assigning unit 136 may assign position information LI1 of the structure ST (structure ST determined in step S803) to the image portion of the extracted unassigned reference station image BG. The assigned position information LI1 corresponds to first position information (absolute coordinates) based on the accurate position information of the reference station 30 included in the reference station image BG.

[0115] If there is no item to which location information has not been assigned (step S802; No), the assigning unit 136 may end the process.

[0116] Here, Fig. 9 shows a specific example in which position information LI1 is assigned to the reference station image BG by the pre-processing (PR1, PR2) described in Fig. 7 and Fig. 8. Fig. 9 explains a specific example in which position information LI1 is assigned using an enlarged portion of the reference station image BG1 shown in Fig. 6(b). The reference station image BG1 shown in Fig. 9 includes the reference station 31 (an example of the reference station 30) and a structure ST1 (an example of a structure ST located in the vicinity of the reference station 30) on which the reference station 31 is installed.

[0117] 9, the accurate position information of the reference station 31 is assumed to be absolute coordinates (X0, Y0, Z0). In this state, in step S703, the assigning unit 136 may calculate position information LI1 of the structure ST1 based on the absolute coordinates (X0, Y0, Z0) and an installation drawing including structural information of the structure ST1. For example, the assigning unit 136 may calculate position information LI1 of the structure ST1 based on the absolute coordinates (X0, Y0, Z0) and information on the length, width, and height of the structure ST.

[0118] Here, it is assumed that the assigning unit 136 extracted the reference station image BG1 as a reference station image BG to which position information has not been assigned in step S802. In step S803, the assigning unit 136 determines that the structure ST in which the reference station 31 included in the reference station image BG1 is installed is the structure ST1.

[0119] As a result, in step S804, the assigning unit 136 assigns the position information LI1 of the structure ST1 to the image portion of the structure ST1 in the reference station image BG1. Fig. 9 shows an example in which the assigning unit 136 assigns absolute coordinates (X1, Y1, Z1), absolute coordinates (X2, Y2, Z2), absolute coordinates (X3, Y3, Z3), and absolute coordinates (X4, Y4, Z4) based on the absolute coordinates (X0, Y0, Z0) of the reference station 31 as the position information LI1 of the structure ST1.

[0120] FIG. 9 shows an example in which the assigning unit 136 assigns the position information LI1 of the structure ST1 in which the reference station 31 is installed to the reference station image BG1.

[0121] However, the reference station image BG1 includes various objects OB (for example, other buildings, parking lots, roads, etc.) in addition to the structure ST1. Therefore, the assigning unit 136 may also assign position information LI1 to objects OB other than the structure ST1 in which the reference station 31 is installed. For example, the assigning unit 136 can calculate the position information LI1 of each object OB in the vicinity of the structure ST1 based on the positional relationship (which can be determined from the installation drawing) between the structure ST1 in which the reference station 31 is installed and the objects OB in the vicinity of the structure ST1, and the position information LI1 of the structure ST1.

[0122] In addition, Figure 9 shows an example in which position information LI1 (first position information based on the accurate position information of the reference station 30) of an object OB contained in one reference station image BG, called the reference station image BG1, is assigned.

[0123] However, the assigning unit 136 may combine multiple reference station images BG and assign position information LI1 of objects OB included in each of the multiple reference station images BG to each of the multiple reference station images BG. For example, the assigning unit 136 may calculate position information LI1 of structures ST included in each reference station image BG based on accurate position information of the reference stations 30 included in each of the reference station images BG that overlap at least a portion of the area AR1. Then, the assigning unit 136 may further calculate position information LI1 of objects OB that exist between the structures ST, for example, based on the position information LI1 calculated for each structure ST included in each reference station image BG.

[0124] The following describes the operation procedures of the information processing device 100 for the absolute coordinate assignment process and the 3D conversion process. As shown in FIG. 6(a), the SAR satellite 10 may observe the same observation range at a predetermined interval while orbiting the Earth along an orbit TR, and transmit an SAR image SR of the observation results to the information processing device 100 each time it captures an image. Therefore, the information processing device 100 may sequentially acquire SAR images SG for each observation range by sequential SAR observation according to the orbits of the SAR satellite 10. The process described below is assumed to be for range R1 (FIG. 6(a)) among the observation ranges, but similar processes are also performed for the other observation ranges.

[0125] Furthermore, the absolute coordinate assignment process and the 3D rendering process may be performed in real time in accordance with the orbit of the SAR satellite 10.

[0126] (6-3. Absolute coordinate assignment processing) In the absolute coordinate assignment process, a pseudo optical image CL_SG is generated by colorizing the grayscale SAR image SG, and absolute coordinates are assigned to the pseudo optical image CL_SG by feature point matching. Hereinafter, the operation procedures of the information processing device 100 in the colorization process in the absolute coordinate assignment process and the feature point matching in the paired coordinate assignment process will be described.

[0127] (6-3-1. Colorization processing) 10 is a flowchart showing the steps of the colorization process. The acquisition unit 131 may determine whether or not it has been able to acquire a SAR image SG from the SAR satellite 10 (step S1001). For example, the acquisition unit 131 may determine whether or not it has been able to acquire a SAR image SG1 from the SAR satellite 10, in which a range R1 was observed by the SAR satellite 10.

[0128] Since the SAR satellite 10 can sequentially acquire SAR images SG1 in which the range R1 is photographed at a rate of 6 images per hour (144 images per 24 hours), the acquisition unit 131 can determine that the SAR image SG1 in which the range R1 is observed has been acquired from the SAR satellite 10 in accordance with the sequential acquisition at this rate.

[0129] While the acquisition unit 131 has not been able to acquire the SAR image SG1 in which the range R1 is observed from the SAR satellite 10 (step S1001; No), the acquisition unit 131 may wait until the SAR image SG1 in which the range R1 is observed can be acquired from the SAR satellite 10.

[0130] On the other hand, if a SAR image SG1 in which range R1 was observed has been acquired from the SAR satellite 10 (step S1001; Yes), the first generation unit 133 may input the currently acquired SAR image SG1 (SAR image SG1 obtained by observing range R1) into the model M1 (step S1002). The currently acquired SAR image SG1 is an original SAR image SG1 that is not colorized and does not have absolute coordinates (SAR image SG1 to which absolute coordinates are to be assigned).

[0131] Then, the first generating unit 133 may generate a pseudo optical image CL_SG1 in which an optical image corresponding to the original SAR image SG1 is pseudo-reproduced based on the output result of the model M1 (step S1003).

[0132] (6-3-2. Feature point matching) 11 is a flowchart showing the procedure for feature point matching. The acquisition unit 131 may determine whether or not a pseudo optical image CL_SG has been generated (step S1101). If a pseudo optical image CL_SG has not been generated (step S1101; No), the acquisition unit 131 may wait until a pseudo optical image CL_SG is generated.

[0133] On the other hand, if a pseudo optical image CL_SG has been generated (step S1101; Yes), the acquisition unit 131 may acquire the generated pseudo optical image CL_SG (step S1102). As shown in FIG. 10, if a pseudo optical image CL_SG1 has been generated by the first generation unit 133, the acquisition unit 131 may determine in step S1102 that a pseudo optical image CL_SG has been generated, and acquire the pseudo optical image CL_SG1 as the generated pseudo optical image CL_SG. The following processing will be described using the pseudo optical image CL_SG1 as an example of the pseudo optical image CL_SG acquired in step S1102.

[0134] The extraction unit 134 may determine information about the area AR2 indicated by the pseudo optical image CL_SG1 (step S1103). For example, the extraction unit 134 may determine information about the area AR2 indicated by the pseudo optical image CL_SG1 based on approximate position information included in the pseudo optical image CL_SG1. For example, the extraction unit 134 may obtain a determination result that the area AR2 indicated by the pseudo optical image CL_SG1 is data resulting from observation of the range R1.

[0135] Therefore, the extraction unit 134 may determine whether the reference station 30 is included in the area AR2 based on the information about the area AR2 (step S1104). The information processing device 100 may be managed by the business operator that installed the reference station 30, and may manage installation location information in a database that defines what kind of reference station 30 is installed at what location in each area. Therefore, the extraction unit 134 can determine whether the reference station 30 is included in the area AR2 by comparing the information about the area AR2 with the installation location information.

[0136] If the reference station 30 is not included in the area AR2 (step S1104; No), the extraction unit 134 may return the processing to step S1101 so that feature point matching can be performed on other pseudo optical images CL_SG that include the reference station 30.

[0137] On the other hand, if the reference station 30 is included in the area AR2 (step S1104; Yes), the extraction unit 134 may acquire a reference station image BG corresponding to the pseudo optical image CL_SG1 based on the information about the area AR2 (step S1105). Because the area AR2 indicated by the pseudo optical image CL_SG1 is data resulting from observation of the range R1, the extraction unit 134 may acquire a reference station image BG in which at least a portion of the range R1 is captured. This reference station image BG is image data to which first position information (position information LI1) has been assigned by pre-processing (PR1, PR2). In the following description, it is assumed that the extraction unit 134 has acquired a reference station image BG1 as a reference station image BG in which at least a portion of the range R1 is captured.

[0138] The extraction unit 134 may use the model M2 to extract a feature point F1 in the area AR1 indicated by the reference station image BG1, and may also extract a feature point F2 in the area AR2 indicated by the pseudo optical image CL_SG1 (step S1106).

[0139] Then, the matching unit 135 may perform matching of the feature points F common to the reference station image BG1 and the pseudo optical image CL_SG1 based on the feature points F1 and F2 (step S1107). Specifically, the matching unit 135 may calculate norms as feature amounts of the feature points F1 and F2, and may match the feature points F between the reference station image BG1 and the pseudo optical image CL_SG1, i.e., between the feature points F1 and F2, that have the smallest distance to each other, as the common feature points F.

[0140] The assigning unit 136 may assign position information LI2 to the pseudo optical image CL_SG1 based on the feature point F2 that matches with the feature point F1 that indicates the object OB (e.g., structure ST1) for which the position information LI1 was calculated, and the position information LI1 (step S1108). For example, the assigning unit 136 may assign position information LI2 to the pseudo optical image CL_SG1 based on a feature point pair that matches between the feature point F1 and the feature point F2, the feature point F2 that matches with the feature point F1 that indicates the object OB, and the position information LI1 assigned to the image portion of the object OB in the reference station image BG1. For example, the assigning unit 136 may perform information transfer, transferring second position information to the image portion of all feature points F2 included in the pair in the pseudo optical image CL_SG1. The assigned position information LI2 is second position information according to the position information LI1 (first position information), and therefore can be considered absolute position information based on the accurate position information of the reference station 30.

[0141] As a result, the acquisition unit 131 acquires a pseudo optical image CL_SG1' to which the position information LI2 has been assigned (step S1109). The pseudo optical image CL_SG1' to which the position information LI2 has been assigned is image data obtained by colorizing the original SAR image SG1 and further assigning absolute coordinates.

[0142] (6-4.3D processing) Fig. 12 is a flowchart (1) showing the procedure of the 3D conversion process. The 3D conversion process may be executed in real time in accordance with the orbit of the SAR satellite 10, following the absolute coordinate assignment process shown in Fig. 11. Fig. 12 shows a scene in which SD conversion is performed based on the pseudo optical image CL_SG1' to which absolute coordinates have been assigned.

[0143] The second generating unit 137 may acquire an original SAR image SG corresponding to the pseudo optical image CL_SG1′ to which absolute coordinates have been assigned (step S1201). Specifically, the second generating unit 137 may acquire an original SAR image SG1.

[0144] For example, the second generation unit 137 may input the original SAR image SG1 to the model M3 (step S1202a). Then, the second generation unit 137 may detect the region of the object OB in the area AR2 indicated by the original SAR image SG1 based on the output result output by the model M3 (step S1203a), and obtain object region information OB_DA2, which is information on the detected region (step S1204a).

[0145] Furthermore, the second generation unit 137 may input the original SAR image SG1 to the model M4 (step S1202b). Then, the second generation unit 137 may estimate the height of the object OB in the area AR2 indicated by the original SAR image SG1 based on the output result output by the model M4 (step S1203b), and obtain object height information OB_DA3, which is information on the estimated height (step S1204b).

[0146] Then, the second generating unit 137 may generate object information OB_DA including object region information OB_DA2 and object height information OB_DA3 (Step S1205).

[0147] Furthermore, the second generation unit 137 may synthesize an original SAR image OB_SG1 having object information OB_DA by associating object information OB with the original SAR image SG1 (step S1206). The original SAR image OB_SG1 has object information OB_DA but does not have absolute coordinates.

[0148] Fig. 13 explains the 3D conversion process that is executed following the process of Fig. 12. Fig. 13 is a flowchart (2) showing the procedure of the 3D conversion process.

[0149] The acquisition unit 131 acquires a pseudo optical image CL_SG1' to which absolute coordinates have been assigned (step S1301).

[0150] The extraction unit 134 extracts feature points F in the area AR2 indicated by the original SAR image OB_SG1 and the area AR2 indicated by the pseudo optical image CL_SG'. Specifically, the extraction unit 134 uses the model M2 to extract feature points F3 that are feature points on the original SAR image OB_SG1 side in the area AR2, and also extracts feature points F4 that are feature points on the pseudo optical image CL_SG1' side in the area AR2 (step S1302).

[0151] Then, the matching unit 135 may perform matching of the feature point F common to the original SAR image OB_SG1 and the pseudo optical image CL_SG1' based on the feature point F3 and the feature point F4 (step S1303). Specifically, the matching unit 135 may calculate a norm as the feature amount of each of the feature point F3 and the feature point F4, and may match the feature point F between the original SAR image OB_SG1 and the pseudo optical image CL_SG1', that is, between the feature point F3 and the feature point F4, such that the distance between them is the smallest, as the common feature point F.

[0152] The original SAR image OB_SG1 is grayscale, and the pseudo-optical image CL_SG1' is color, but since both are SAR images based on the same imaging principle, they can be matched appropriately.

[0153] The assigning unit 136 may assign object information OB_DA to the pseudo optical image CL_SG1' based on the feature point pair matched between the feature point F3 and the feature point F4 and the object information OB_DA associated with the image portion of the paired feature point F3 in the original SAR image OB_SG1 (step S1304). For example, the assigning unit 136 may perform information transfer to transfer the object information OB_DA to image portions of all feature points F4 included in the pair in the pseudo optical image CL_SG1'.

[0154] As a result, the acquisition unit 131 acquires a pseudo optical image CL_SG1" to which the object information OB_DA has been assigned (step S1305). The pseudo optical image CL_SG1" to which the object information OB_DA has been assigned is image data in which the original SAR image SG1 has been colorized, absolute coordinates have been assigned, and information that serves as the basis for 3D conversion has been assigned.

[0155] In this state, the second generation unit 137 may generate a three-dimensional model MD in which the object OB, which is expressed in a plane in the pseudo optical image CL_SG1", is expressed in a three-dimensional manner, based on the object information OB_DA assigned to the pseudo optical image CL_SG1" (step S1306). Note that the second generation unit 137 may generate the final three-dimensional model MD' by processing the three-dimensional model MD to make it closer to the actual color according to the RGB elements included in the pseudo optical image CL_SG1".

[0156] (6-5. Services using 3D models) As a service that utilizes a three-dimensional model, the operation procedure of the information processing device 100 in support processing for dealing with landslides will be described. The support processing may include a calculation process for calculating the volume of sediment and a control process for removing sediment, and the operation procedure of the information processing device 100 in each process will be described below.

[0157] (6-5-1. Calculation process) 14 is a flowchart showing the procedure for calculating the sediment volume. When a critical natural phenomenon (e.g., an earthquake) occurs while the three-dimensional model MD is being generated, the acquisition unit 131 may acquire 3D models MD1 and MD2 as the three-dimensional models MD before and after the occurrence of the critical natural phenomenon is detected (step S1401).

[0158] The 3D model MD1 may be a 3D model MD of the ground surface of a given area in normal times before a critical natural phenomenon occurs, and the 3D model MD2 may be a 3D model MD of the ground surface of the same area after a critical natural phenomenon occurs.

[0159] The matching unit 135 may match the relative positional relationship between the 3D model MD1 and the 3D model MD2 (step S1402).

[0160] The estimation unit 138 may calculate the difference between the topography before the occurrence of the critical natural phenomenon and the topography after the occurrence of the critical natural phenomenon based on the matching result (step S1403).

[0161] Furthermore, the estimation unit 138 may detect the soil and sand portion based on the difference calculated in step S1403 (step S1404). For example, the estimation unit 138 may detect the portion corresponding to the difference (i.e., the portion that has changed between the topography before the occurrence of the critical natural phenomenon and the topography after the occurrence of the critical natural phenomenon) as the soil and sand portion.

[0162] Then, the estimation unit 138 may calculate the volume of soil corresponding to the soil portion based on the three-dimensional model MD corresponding to the soil portion (step S1405).

[0163] The estimation unit 138 may further detect whether a landslide disaster has occurred based on the calculated sediment volume. For example, the estimation unit 138 may determine whether the sediment volume calculated in step S1405 exceeds a threshold value (step S1406). If the sediment volume does not exceed the threshold value (step S1406; No), the processing may be terminated.

[0164] On the other hand, if the volume of sediment calculated in step S1405 exceeds the threshold value (step S1406; Yes), the estimation unit 138 may detect that a landslide disaster has occurred.

[0165] (6-5-2. Control process for soil removal) 15 is a flowchart showing the procedure of the control process for removing sediment. When detecting the occurrence of a landslide disaster, the estimation unit 138 may estimate the type and number of equipment required to remove the sediment portion based on the volume of the sediment portion (step S1501).

[0166] The notification unit 139 may perform control so that information based on the estimation result is notified to the user U (step S1502).

[0167] <Modification> The information processing according to the embodiment may be implemented in a manner different from the above example, and therefore, a modified example of the information processing according to the embodiment will be described below.

[0168] In the above embodiment, an example was shown in which the information processing device 100 assigns first position information, which is absolute position information (absolute coordinates) based on the accurate position information of the reference station 30, to the reference station image BG (optical image) in advance, and assigns second position information corresponding to the first position information to the pseudo optical image CL_SG by performing feature point matching between the reference station image BG and the pseudo optical image CL_SG that includes the reference station 30. In such a method, pre-processing is required to assign the first position information to the reference station image BG, and the pseudo optical image CL_SG used for feature point matching needs to include the reference station 30.

[0169] Therefore, a modified example will be described in which a method is provided that does not require pre-processing to assign first position information to the reference station image BG, and that allows feature point matching even for a pseudo optical image CL_SG that does not include the reference station 30. The information processing according to the modified example will be described in detail below.

[0170] [1. Overall view of the modified example] First, an overview of the modified example will be described using Fig. 16. Fig. 16 is a diagram showing an overview of the modified example. Fig. 16 corresponds to the example of Fig. 2, but is partially different due to the information processing according to the modified example. Differences between the information processing according to the modified example and the information processing according to the embodiment shown in Fig. 2 will be described.

[0171] 2 shows an example in which first position information is assigned to the reference station image BG based on the accurate position information of the reference station 30. Specifically, in the absolute coordinate assignment process, the position information of an object in the vicinity of the reference station 30 that is included in the area AR1 is calculated as the first position information, which is based on the accurate position information of the reference station 30, and the first position information is assigned to the image portion of the object in the reference station image BG.

[0172] In this type of absolute coordinate assignment process, pre-processing is required to assign first position information to the reference station image BG, and the pseudo optical image CL_SG used for feature point matching must include the reference station 30. In other words, the reference station 30 must be included in the original SAR image SG.

[0173] Therefore, the inventors of the present invention have come up with a method of using an aerial photograph AP taken by an aircraft 70 with RTK positioning function instead of the reference station image BG, which eliminates the need for pre-processing to assign first position information to the reference station image BG and enables feature point matching even in pseudo-optical images CL_SG that do not include the reference station 30.

[0174] 16, the information processing system Sy includes an air vehicle 70 instead of the optical satellite 60. The air vehicle 70 may be, for example, an unmanned air vehicle such as a drone, or a small airplane piloted by a person.

[0175] The flying object 70 may have a computing function for calculating its own accurate position information by RTK positioning and a photographing function. The computing function here may be the receiving device 20. That is, the flying object 70 is equipped with the receiving device 20 and acquires aerial photographs AP by taking photographs from the sky with a camera.

[0176] For this reason, the aerial photograph AP has accurate position information (absolute coordinates). Specifically, the aerial photograph AP has position information based on the accurate position information of the flying object 70, and the position information in the area AR1 indicated by the aerial photograph AP is assigned as the first position information at the time of photographing. For example, the photographing function of the flying object 70 may cooperate with the receiving device 20 to acquire the aerial photograph AP, which has position information based on the accurate position information of the flying object 70 and has the position information in the area AR1 indicated by the aerial photograph AP assigned as the first position information.

[0177] Furthermore, the aerial photograph AP may be taken in advance by flying an aircraft 70 over the entire Japanese sky at any time, and then uploaded to the information processing device 100. However, since the aerial photograph AP is a full-color optical image, it may not be possible to accurately match feature points between the aerial photograph AP and the SAR image.

[0178] For this reason, in the absolute coordinate assignment process, which is executed in real time as the SAR satellite 10 orbits, the SAR image SG is colorized to generate a pseudo optical image CL_SG, which is a full-color image (i.e., an RGB image) that approximates the color of the optical image. Also, in the absolute coordinate assignment process, second position information corresponding to the first position information is assigned to the pseudo optical image CL_SG, which does not have absolute coordinates, by matching feature points between the pseudo optical image CL_SG and the aerial photograph AP, resulting in a pseudo optical image CL_SG'. The second position information here can be said to be absolute position information (absolute coordinates) based on the accurate position information of the flying object 70.

[0179] 2. Feature Point Matching in Modifications Next, a specific operation procedure of the information processing device 100 in feature point matching according to the modified example will be described. Fig. 17 is a flowchart showing the procedure of feature point matching according to the modified example. Note that in the example of Fig. 17, it is assumed that the information processing device 100 has acquired the original SAR image SG1, which is the SAR image SG1 in which the range R1 was observed by the SAR satellite 10, and has therefore performed colorization of the original SAR image SG1 using the model M1. That is, in the example of Fig. 17, it is assumed that the information processing device 100 has generated a pseudo optical image CL_SG1 in which an optical image corresponding to the original SAR image SG1 is pseudo-reproduced.

[0180] First, the acquiring unit 131 may determine whether or not a pseudo optical image CL_SG has been generated (step S1701). If a pseudo optical image CL_SG has not been generated (step S1701; No), the acquiring unit 131 may wait until a pseudo optical image CL_SG is generated.

[0181] On the other hand, if the pseudo optical image CL_SG has been generated (step S1701; Yes), the acquiring unit 131 may acquire the generated pseudo optical image CL_SG (step S1702). Because the pseudo optical image CL_SG1 has already been generated by the first generating unit 133, the acquiring unit 131 may determine in step S1702 that the pseudo optical image CL_SG has been generated, and may acquire the pseudo optical image CL_SG1 as the generated pseudo optical image CL_SG. The following processing will be described using the pseudo optical image CL_SG1 as an example of the pseudo optical image CL_SG acquired in step S1702.

[0182] The extraction unit 134 may determine information about the area AR2 indicated by the pseudo optical image CL_SG1 (step S1703). For example, the extraction unit 134 may determine information about the area AR2 indicated by the pseudo optical image CL_SG1 based on approximate position information included in the pseudo optical image CL_SG1. For example, the extraction unit 134 may obtain a determination result that the area AR2 indicated by the pseudo optical image CL_SG1 is data resulting from observing the range R1.

[0183] The extraction unit 134 may acquire an aerial photograph AP corresponding to the pseudo optical image CL_SG1 based on the information of the area AR2 (step S1704). Because the area AR2 indicated by the pseudo optical image CL_SG1 is data resulting from observation of the range R1, the extraction unit 134 may acquire an aerial photograph AP in which at least a portion of the range R1 is captured. In the following description, it is assumed that the extraction unit 134 acquires an aerial photograph AP1 as the aerial photograph AP in which at least a portion of the range R1 is captured.

[0184] The extraction unit 134 may match the coordinate systems of the aerial photograph AP1 and the pseudo optical image CL_SG1 (step S1705). For example, the extraction unit 134 may perform geographic coordinate transformation to match the coordinate systems of the aerial photograph AP1 and the pseudo optical image CL_SG1.

[0185] The extraction unit 134 may also match the image scale between the aerial photograph AP1 and the pseudo optical image CL_SG1 (step S1706). For example, the flying altitudes of the flying object 70 and the SAR satellite 10 are different, and therefore the image scales of the aerial photograph AP1 and the pseudo optical image CL_SG1 are different. For this reason, the extraction unit 134 may match the image scales by, for example, downsizing the resolution of the aerial photograph AP1.

[0186] In this state, the extraction unit 134 may use the model M2 to extract a feature point F1 in the area AR1 indicated by the aerial photograph AP1, and may also extract a feature point F2 in the area AR2 indicated by the pseudo optical image CL_SG1 (step S1707).

[0187] The matching unit 135 may then match the feature points F common to the aerial photograph AP1 and the pseudo optical image CL_SG1 based on the feature points F1 and F2 (step S1708). Specifically, the matching unit 135 may calculate norms as feature amounts for the feature points F1 and F2, and match the feature points F between the aerial photograph AP1 and the pseudo optical image CL_SG1, i.e., between the feature points F1 and F2, that have the smallest distance to each other, as the common feature points F.

[0188] The assigning unit 136 may assign position information LI2 to the pseudo optical image CL_SG based on the feature point pair that is matched between the feature point F1 and the feature point F2 and the position information LI1 assigned to the image portion of the paired feature point F1 in the aerial photograph AP1 (step S1709). For example, the assigning unit 136 may also transfer the position information LI2 on a pixel-by-pixel basis using the position information LI2 assigned to the image portions of all feature points F2 included in the pair in the pseudo optical image CL_SG1.

[0189] As a result, the acquisition unit 131 acquires a pseudo optical image CL_SG1' to which the position information LI2 has been assigned (step S1710). The pseudo optical image CL_SG1' to which the position information LI2 has been assigned is image data obtained by colorizing the original SAR image SG1 and further assigning absolute coordinates.

[0190] In this way, the information processing according to the modified example does not require pre-processing to assign first position information, and feature point matching can be performed even for pseudo-optical images CL_SG that do not include the reference station 30, thereby enabling feature point matching to be performed more efficiently.

[0191] <Hardware configuration> The information processing device 100 described above may be realized, for example, by a computer 1000 configured as shown in Fig. 18. Fig. 18 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0192] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 starts up, programs that depend on the hardware of the computer 1000, and the like.

[0193] The HDD 1400 stores programs executed by the CPU 1100, data used by these programs, etc. The communication interface 1500 receives data from other devices via a predetermined communication network and sends the data to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0194] The CPU 1100 controls an output device such as a display and an input device such as a keyboard via the input / output interface 1600. The CPU 1100 acquires data from the input device via the input / output interface 1600. The CPU 1100 also outputs generated data to the output device via the input / output interface 1600.

[0195] Media interface 1700 reads a program or data stored in recording medium 1800 and provides it to CPU 1100 via RAM 1200. CPU 1100 loads the program or data from recording medium 1800 onto RAM 1200 via media interface 1700 and executes the loaded program. Recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0196] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 130. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via a predetermined communication network.

[0197] <Other> Furthermore, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0198] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0199] Furthermore, the above-described embodiments can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0200] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms that include the aspects described in the "present invention" section and that have been modified and improved in various ways based on the knowledge of those skilled in the art. [Explanation of symbols]

[0201] Sy Information Processing System 1. SAR observation system 2. RTK positioning system 10 SAR satellites 20 Receiving device 30 Reference station 40 GNSS satellites 50 Ascension Core System 60 optical satellite 70 Flying Objects 100 Information processing device 120 Storage section 130 Control Unit 131 Acquisition Department 132 Learning Department 133 1st generation part 134 Extraction part 135 Matching Department 136 Granting Department 137 Second generation part 138 Estimation Department 139 Notification Department

Claims

1. a first generation unit that generates an RGB image corresponding to a predetermined synthetic aperture radar image based on the predetermined synthetic aperture radar image and a predetermined machine learning model; an acquisition unit that acquires a predetermined optical image corresponding to the predetermined synthetic aperture radar image, the optical image having first position information assigned thereto, the first position information being absolute position information based on predetermined position information; an extractor that extracts feature points in a first area represented by the optical image and a second area represented by the RGB image; a matching unit that matches the feature points between the first area and the second area; an assigning unit that assigns second position information, which is absolute position information corresponding to the first position information, to the RGB image based on the matching result; An information processing system comprising:

2. The first generation unit a combination of a sample SAR image and a sample optical image is used as training data as a combination of images having similar photographing areas, and the predetermined synthetic aperture radar image is input to a machine learning model that has been trained to simulate an optical image corresponding to the input synthetic aperture radar image, thereby generating the RGB image in which the optical image corresponding to the predetermined synthetic aperture radar image is simulated; The information processing system according to claim 1 .

3. Further including a reference station with known location information; the acquisition unit acquires, as the optical image, a satellite image that is an image including the reference station, and to which the first position information based on position information of the reference station is assigned. The information processing system according to claim 1 .

4. the assigning unit calculates, as the first position information, position information of an object that is included in the first area and is located around the reference station, the position information being based on the position information of the reference station, and assigns the first position information to an image portion of the object in the satellite image. The information processing system according to claim 3 .

5. the extraction unit extracts first feature points that are feature points in the first area based on the satellite image, and extracts second feature points that are feature points in the second area based on the RGB image; the matching unit matches the feature points common to the first feature points and the second feature points; the assigning unit assigns the second position information, which is absolute position information, to the RGB image by transferring absolute position information corresponding to the first position information to the entire second area based on the first position information and second feature points that match with the first feature points that indicate the object for which the first position information has been calculated, among the second feature points. The information processing system according to claim 4 .

6. The aircraft further includes a computing function for calculating its own position information by RTK (Real Time Kinematic) positioning and a photographing function, The acquisition unit acquires, as the optical image, an aerial photograph taken from the sky by the photographing function, to which the first position information based on the position information of the flying object calculated by the calculation function is assigned. The information processing system according to claim 1 .

7. the extraction unit extracts first feature points that are feature points in the first area based on the aerial photograph, and extracts second feature points that are feature points in the second area based on the RGB image; the matching unit matches the feature points common to the first feature points and the second feature points; the assigning unit assigns the second position information, which is absolute position information, to the RGB image by transferring absolute position information corresponding to the first position information to the entire second area based on the second feature points matched with the first feature points and the first position information assigned to the image portion of the aerial photograph at the first feature points. The information processing system according to claim 6.

8. a second generation unit that generates a three-dimensional model that virtually represents a state of the second area, the three-dimensional model having the second position information, based on the predetermined synthetic aperture radar image and the RGB image to which the second position information is assigned; Further preparations The information processing system according to claim 1 .

9. the second generation unit detects an object region from the second area based on the predetermined synthetic aperture radar image and a predetermined machine learning model, estimates a height of the object indicated by the region, and associates object information including at least the region and the height with the predetermined synthetic aperture radar image, thereby synthesizing the predetermined synthetic aperture radar image having the object information. The information processing system according to claim 8 .

10. the extraction unit extracts a third feature point that is a feature point in the second area based on the predetermined synthetic aperture radar image having the object information, and extracts a fourth feature point that is a feature point in the second area based on the RGB image to which the second position information is assigned; the matching unit matches the feature points common to the third feature points and the fourth feature points; the assigning unit assigns the object information to the image portion of the fourth feature point in the RGB image to which the second position information has been assigned, based on the fourth feature point matched with the third feature point and the object information associated with the image portion of the third feature point in the RGB image having the object information. The information processing system according to claim 9 .

11. the second generation unit generates the three-dimensional model from the RGB image to which the object information is assigned. The information processing system according to claim 10.

12. An information processing method executed by an information processing system, a first generation step of generating an RGB image corresponding to a predetermined synthetic aperture radar image based on the predetermined synthetic aperture radar image and a predetermined machine learning model; an acquisition step of acquiring a predetermined optical image corresponding to the predetermined synthetic aperture radar image, the optical image being assigned first position information which is absolute position information based on predetermined position information; an extraction step of extracting feature points in a first area represented by the optical image and a second area represented by the RGB image; a matching step of matching the feature points between the first area and the second area; an assigning step of assigning second position information, which is absolute position information corresponding to the first position information, to the RGB image based on the matching result; An information processing method including:

Images