A Remote Sensing Image Simulation Method Based on Neural Radiance Fields

Abstract

The present invention provides a remote sensing image simulation method based on neural radiance fields, belonging to the field of remote sensing image processing. It includes: collecting multi-view remote sensing images to obtain image data of the scene and target that need to be simulated with multi-view images; constructing a 3D reconstruction network, using a convolutional neural network to extract features from the remote sensing images obtained after data collection, and using a multi-layer perceptron to build a 3D reconstruction network; constructing a volume rendering model, establishing a visible light transmission equation and an infrared transmission equation, and rendering visible light images and infrared images; training the image simulation model and updating the parameters of the image simulation model; inputting the imaging position and imaging perspective of the image to be simulated into the trained image simulation model to generate a simulated image. The present invention can achieve the function of 3D reconstruction of remote sensing scenes under the condition of few views, and can simultaneously obtain visible light simulated images and infrared simulated images.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing image processing, and more particularly relates to a remote sensing image simulation method based on neural radiance fields. Background Art

[0002] Traditional image simulation methods include two steps: 3D reconstruction and image rendering. 3D reconstruction usually requires using specialized 3D modeling computer-aided design software to perform vector model modeling or voxel model modeling on targets, scenes, and maps. At the same time, a complete material library needs to be established to represent the texture, color, and transparency of 3D models. However, the establishment of 3D modeling and the material library requires a large amount of manpower and time, and cannot be adapted to application scenarios with strong timeliness requirements such as disaster assessment and damage assessment. Image rendering requires the use of a specialized image renderer and setting sensor parameters to render each pixel point of the image relying on ray tracing. Phenomena such as pixel holes, resolution degradation, and detail loss will occur in the rendered image. Summary of the Invention

[0003] The purpose of the present invention is to overcome the drawbacks of traditional image simulation methods in the 3D reconstruction and image rendering steps, give full play to the feature expression ability of neural networks, and improve the automation degree of image simulation. A remote sensing image simulation method based on neural radiance fields is proposed.

[0004] The technical solution adopted by the present invention is as follows:

[0005] A remote sensing image simulation method based on neural radiance fields, comprising the following steps:

[0006] Step 1, multi-view remote sensing image acquisition: Using an optical remote sensing satellite and a rotor UAV platform to collect data on the scene and target that need to be simulated with multi-view images;

[0007] Step 2, 3D reconstruction network construction: Using a convolutional neural network to extract features from the remote sensing images obtained after data collection, using a multi-layer perceptron to build a 3D reconstruction network, taking the features extracted by the convolutional neural network, the 3D positions of image acquisition, and the 2D azimuth views of image acquisition as the inputs of the 3D reconstruction network, and the outputs of the 3D reconstruction network being color, voxel density, and apparent temperature;

[0008] Step 3, volume rendering model construction: Establishing a visible light transfer equation and an infrared transfer equation, taking color and voxel density as the inputs of the visible light transfer equation, and outputting a visible light rendered image, taking voxel density and apparent temperature as the inputs of the infrared transfer equation, and outputting an infrared rendered image;

[0009] Step 4, Image Simulation Model Training: Input the multi-view remote sensing images, three-dimensional positions, and two-dimensional azimuth perspectives collected into the three-dimensional reconstruction network model. Use the output of the three-dimensional reconstruction network model as the input of the volume rendering model. Calculate the loss function between the image output by the volume rendering model and the real images collected. Use the optimization algorithm to minimize the loss function and train and update the parameters in the three-dimensional reconstruction network. After the training converges, obtain the image simulation model;

[0010] Step 5, Multi-view Image Simulation: Input the imaging position and imaging perspective of the image to be simulated into the trained image simulation model to generate a simulated image. Set the imaging position and imaging perspective to be simulated to generate multi-view simulated images.

[0011] Further, the optical remote sensing satellite in Step 1 is used to capture visible light remote sensing images and infrared remote sensing images with a ground resolution of 0.3 meters. The rotor UAV platform is used to capture optical remote sensing images and infrared remote sensing images with a ground resolution of 0.1 meters. The data collection of the scene and the target includes 10 different pitch angles and 10 different azimuth angles.

[0012] Further, the convolutional neural network in Step 2 is ResNet18. The three-dimensional reconstruction network includes three multi-layer perceptrons. Each multi-layer perceptron includes 5 hidden layers. The first multi-layer perceptron encodes the three-dimensional position and image features of the image collection to obtain the apparent temperature. The second multi-layer perceptron encodes the three-dimensional position and image features of the image collection to obtain the voxel density and a latent variable. The third multi-layer perceptron encodes the latent variable, the two-dimensional azimuth perspective of the image collection, and the image features to obtain the color.

[0013] Further, the visible light transfer equation in Step 3 is for the visible light band of 380nm - 760nm, and the infrared transfer equation is for the mid-infrared band of 3 - 5μm. Both the visible light transfer equation and the infrared transfer equation are based on the linear transfer theory of geometric optics and do not consider the interference and diffraction of light waves.

[0014] Further, the input data of the image simulation model in Step 4 are images of the same scene taken at different three-dimensional positions and different two-dimensional azimuth perspectives, as well as the three-dimensional position and the two-dimensional azimuth perspective; among them, the three-dimensional position and the two-dimensional azimuth perspective do not distinguish between the satellite platform and the UAV platform. The three-dimensional position only includes 3 distance parameters, and the two-dimensional azimuth perspective only includes 2 angle parameters.

[0015] Further, the loss function in Step 4 uses the image structural similarity function. The image structural similarity function is the weighted sum of the brightness similarity, contrast similarity, gray-scale similarity, texture similarity, and texture similarity between two images. The optimization algorithm is the Adam algorithm.

[0016] Further, when setting the imaging position and imaging perspective described in step 5, the imaging position is preferably set first, and the imaging positions are sequentially set according to the decreasing order of the imaging distance, and the imaging perspectives are sequentially set in the counterclockwise order.

[0017] The advantages of the present invention over the prior art are as follows:

[0018] 1. The present invention can realize the three-dimensional reconstruction function of the remote sensing scene under the condition of few perspectives, reducing the data demand for three-dimensional reconstruction;

[0019] 2. The present invention can render the scene through unstructured data, reducing the limiting conditions of the data acquisition platform;

[0020] 3. The present invention can obtain visible light simulation images and infrared simulation images simultaneously, expanding the application range of image simulation.

[0021] In summary, the present invention adopts a remote sensing image simulation method based on neural radiance fields, which can implicitly express the three-dimensional models of targets, scenes, and maps, saving the labor cost and time cost of three-dimensional reconstruction, and at the same time can generate more detailed simulation images. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is the overall flowchart of a remote sensing image simulation method based on neural radiance fields in an embodiment of the present invention.

[0023] Figure 2 is the structural schematic diagram of the neural radiance field in an embodiment of the present invention. EMBODIMENTS

[0024] The following further describes the specific embodiments and basic principles of the present invention with reference to the accompanying drawings.

[0025] A remote sensing image simulation method based on neural radiance fields, as Figure 1 shown, includes the following steps:

[0026] Step 1, multi-view remote sensing image acquisition, using an optical remote sensing satellite and a rotor UAV platform to collect data of the scene and target that need to be multi-view image simulated. The ground resolution of the visible light remote sensing image and the infrared remote sensing image taken by the optical remote sensing satellite is 0.3 m, and the ground resolution of the optical remote sensing image and the infrared remote sensing image taken by the rotor UAV platform is 0.1 m. The data acquisition of the scene and target includes 10 different pitch angles and 10 different azimuth angles;

[0027] Step 2, three-dimensional reconstruction network construction, as Figure 2As shown in the figure, a convolutional neural network is used to extract features from the remotely sensed images obtained after data acquisition. A three-dimensional reconstruction network is built using a multi-layer perceptron. The features extracted by the convolutional neural network, the three-dimensional positions of image acquisition, and the two-dimensional azimuth angles of image acquisition are used as the inputs of the three-dimensional reconstruction network. The outputs of the three-dimensional reconstruction network are color, voxel density, and apparent temperature. Among them, the convolutional neural network is ResNet18, and the three-dimensional reconstruction network includes three multi-layer perceptrons. Each multi-layer perceptron includes 5 hidden layers. The first multi-layer perceptron encodes the three-dimensional positions of image acquisition and the image features to obtain the apparent temperature. The second multi-layer perceptron encodes the three-dimensional positions of image acquisition and the image features to obtain the voxel density and a latent variable. The third multi-layer perceptron encodes the latent variable, the two-dimensional azimuth angles of image acquisition, and the image features to obtain the color;

[0028] Step 3: Build a volume rendering model. Establish a visible light transfer equation and an infrared transfer equation. Use color and voxel density as the inputs of the visible light transfer equation to output a visible light rendered image. Use voxel density and apparent temperature as the inputs of the infrared transfer equation to output an infrared rendered image. Among them, the visible light transfer equation is for the visible light band of 380nm - 760nm, and the infrared transfer equation is for the mid-infrared band of 3 - 5μm. Both the visible light transfer equation and the infrared transfer equation are based on the linear transfer theory of geometric optics, without considering the interference and diffraction of light waves;

[0029] Step 4: Train the image simulation model. Input the multi-view remotely sensed images, three-dimensional positions, and two-dimensional azimuth angles collected into the three-dimensional reconstruction network model. Use the outputs of the three-dimensional reconstruction network model as the inputs of the volume rendering model. Calculate the loss function between the images output by the volume rendering model and the real images collected. Use an optimization algorithm to minimize the loss function and train and update the parameters in the three-dimensional reconstruction network. After training converges, an image simulation model is obtained. Among them, the input data of the image simulation model are images of the same scene taken at different three-dimensional positions and different two-dimensional azimuth angles, as well as the three-dimensional positions and two-dimensional azimuth angles. Among them, the three-dimensional positions and two-dimensional azimuth angles do not distinguish between satellite platforms and drone platforms. The three-dimensional positions only include 3 distance parameters, and the two-dimensional azimuth angles only include 2 angle parameters. The loss function uses an image structure similarity function, which is the weighted sum of the brightness similarity, contrast similarity, gray level similarity, texture similarity, and texture similarity between two images. The optimization algorithm is the Adam algorithm;

[0030] Step 5, multi-view image simulation. Input the imaging position and imaging angle of the image to be simulated into the trained image simulation model to generate a simulated image. Set the imaging position and imaging angle to be simulated to generate multi-view simulated images. When setting the imaging position and imaging angle, the imaging position is preferably set first, and the imaging positions are sequentially set in the order of decreasing imaging distance, and the imaging angles are sequentially set in the counterclockwise order.

Claims

1. A remote sensing image simulation method based on neural radiance fields, characterized in that, It includes the following steps: Step 1, multi-view remote sensing image acquisition: Use an optical remote sensing satellite and a rotary-wing UAV platform to collect data on the scene and target for which multi-view image simulation is required; Step 2, three-dimensional reconstruction network construction: Use a convolutional neural network to extract features from the remote sensing images obtained after data collection, and use a multi-layer perceptron to build a three-dimensional reconstruction network. Take the features extracted by the convolutional neural network, the three-dimensional position of image acquisition, and the two-dimensional azimuth view of image acquisition as the input of the three-dimensional reconstruction network. The output of the three-dimensional reconstruction network is color, voxel density, and apparent temperature; Step 3, volume rendering model construction: Establish a visible light transfer equation and an infrared transfer equation. Take color and voxel density as the input of the visible light transfer equation and output a visible light rendered image. Take voxel density and apparent temperature as the input of the infrared transfer equation and output an infrared rendered image; Step 4, image simulation model training: Input the collected multi-view remote sensing images, three-dimensional positions, and two-dimensional azimuth views into the three-dimensional reconstruction network model. Take the output of the three-dimensional reconstruction network model as the input of the volume rendering model. Calculate the loss function between the image output by the volume rendering model and the collected real image. Use an optimization algorithm to minimize the loss function and train and update the parameters in the three-dimensional reconstruction network. After training convergence, obtain an image simulation model; Step 5, multi-view image simulation: Input the imaging position and imaging view of the image to be simulated into the trained image simulation model to generate a simulated image, and set the imaging position and imaging view to be simulated to generate multi-view simulated images.

2. The remote sensing image simulation method based on neural radiance fields according to claim 1, wherein, The optical remote sensing satellite in Step 1 is used to capture visible light remote sensing images and infrared remote sensing images with a ground resolution of 0.3 meters. The rotary-wing UAV platform is used to capture optical remote sensing images and infrared remote sensing images with a ground resolution of 0.1 meters. The data collection of the scene and target includes 10 different pitch angles and 10 different azimuth angles.

3. A remote sensing image simulation method based on neural radiance fields according to claim 1, characterized in that The convolutional neural network mentioned in Step 2 is ResNet18. The three-dimensional reconstruction network includes three multi-layer perceptrons. Each multi-layer perceptron includes 5 hidden layers. The first multi-layer perceptron encodes the three-dimensional position of image acquisition and the image features to obtain the apparent temperature. The second multi-layer perceptron encodes the three-dimensional position of image acquisition and the image features to obtain the voxel density and a latent variable. The third multi-layer perceptron encodes the latent variable, the two-dimensional azimuth view of image acquisition, and the image features to obtain the color.

4. A remote sensing image simulation method based on neural radiance fields according to claim 1, characterized in that, The visible light transfer equation in Step 3 is for the visible light band of 380nm - 760nm, and the infrared transfer equation is for the mid-infrared band of 3 - 5μm. Both the visible light transfer equation and the infrared transfer equation are based on the linear transfer theory of geometric optics and do not consider the interference and diffraction of light waves.

5. A remote sensing image simulation method based on neural radiance fields according to claim 1, characterized in that, The input data of the image simulation model described in step 4 are images of the same scene taken from different three-dimensional positions and different two-dimensional azimuth perspectives, as well as three-dimensional positions and two-dimensional azimuth perspectives; among them, the three-dimensional positions and two-dimensional azimuth perspectives do not distinguish between satellite platforms and unmanned aerial vehicle platforms, the three-dimensional positions only include 3 distance parameters, and the two-dimensional azimuth perspectives only include 2 angle parameters.

6. The remote sensing image simulation method based on neural radiance fields according to claim 1, characterized in that, The loss function described in step 4 adopts an image structural similarity function, and the image structural similarity function is the weighted sum of luminance similarity, contrast similarity, gray-scale similarity, texture similarity, and texture similarity between two images. The optimization algorithm is the Adam algorithm.

7. A remote sensing image simulation method based on neural radiance fields according to claim 1, characterized in that, When setting the imaging position and imaging perspective described in step 5, the imaging position is preferably set first, the imaging positions are set in sequence according to the decreasing order of imaging distance, and the imaging perspectives are set in sequence in the counterclockwise direction.

Images