Three-dimensional remote sensing radiation transfer simulation method based on omniverse platform

Abstract

The present application relates to the technical field of remote sensing data simulation, in particular to a three-dimensional remote sensing radiation transmission simulation method based on an Omniverse platform, which is used for simulating a laser radar and comprises the following steps: importing a three-dimensional scene model of a pre-established test area into the Omniverse platform, and setting ground objects in the three-dimensional scene model as static collision bodies; simulating and connecting the laser radar and a platform on which the laser radar is mounted, and setting a scanning mode of the laser radar; obtaining scanning points of the laser radar through an API interface of the Omniverse platform, combining the collision points of the laser radar with the spatial positions of the platform on which the laser radar is mounted, and generating point cloud data with depth and semantic segmentation information; and saving the synthesized data and displaying each frame of information in real time on a window. The present application popularizes the three-dimensional radiation transmission modeling idea through a visual radiation transmission process, is more popular, and the obtained simulation data set is more complete.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing data simulation, and more particularly to a three-dimensional remote sensing radiation transmission simulation method based on an Omniverse platform. BACKGROUND

[0002] Forestry work itself has the characteristics of vast resources and long growth cycle. Remote sensing technology has become an important tool in the forestry industry because of its characteristics of rapid and large-scale resource investigation. Remote sensing technology has developed from qualitative identification of ground objects relying on experience to quantitative remote sensing of techniques and methods for quantitatively extracting surface parameters in electromagnetic wave signals for earth observation. However, due to the diversity of the ground surface and the complexity of the remote sensing signal radiation transmission process, the problem of remote sensing ill-posed inversion is particularly prominent. Prior knowledge (data set) is an important support for solving the underdetermined problem of physical model inversion, but its completeness, accuracy and timeliness are limited. The main way to obtain data sets for ground observation is usually to establish a large-scale comprehensive remote sensing observation test field, to obtain data sets by using wireless sensor networks and unmanned aerial remote sensing. However, this method has problems such as large investment, limited number, strong spatio-temporal variability, and insufficient physical quantity observation capability. At present, there is still a lack of effective means to integrate relatively discrete observation data to form high-quality prior knowledge to promote the development of mechanism model construction, surface parameter inversion, remote sensing product production and authenticity testing.

[0003] Complete data sets are difficult to obtain through observation, and computer simulation models are an important way to obtain complete data sets. Among them, the three-dimensional remote sensing radiation transmission model has the least assumptions and the highest precision, and is an important method for studying quantitative remote sensing mechanism and improving inversion accuracy, and is in increasing demand.

[0004] The role of computer simulation models is not only to obtain prior knowledge, but also to deepen the understanding of remote sensing mechanism for beginners, and to do spatio-temporal interpolation on discrete data such as satellite data to produce continuous and complete data sets. Some sensitive analysis that is difficult to carry out experiments, such as forest fires and pests, cannot be done in the field, and these can be simulated by computer simulation models to produce simulation data, which can also directly support field experiments and product verification. The data generated by simulation can also be directly used as AI training data, making it easy to train AI in various directions.

[0005] At present, relevant teams at home and abroad have recently developed DART (Discrete Anisotropic Radiative Transfer) (Gastellu-Etchegorry et al., 1996), LESS (Large-Scale remote sensing data and image Simulation) (Qi et al., 2019) and other models, which realize multi-scale, multi-band and multi-temporal simulation of target scenes.

[0006] The LESS model is a three-dimensional real structure radiation transfer model based on ray tracing. It can simulate the transmission process (absorption, reflection and transmission) of incident light in the scene and output corresponding simulation data. The so-called real structure refers to a scene element (for example, trees, houses, etc.) with an arbitrary complex structure represented by a triangular facet. The real structure here is relative to the simplified way of assuming tree crowns as ellipsoids and expressing tree leaves as continuous and uniform media in traditional remote sensing models. It is a three-dimensional radiation transfer model capable of simultaneously performing forward and backward ray tracing (path tracing) and simulating large-scale remote sensing data.

[0007] DART is also a three-dimensional real structure model that simulates complex large scenes. Unlike LESS, it uses a bidirectional path tracing algorithm, which has a great leap in computational efficiency. As a pioneer in three-dimensional radiation transfer models, DART has more functions, such as three-dimensional emission, scattering, interception and absorption of radiation, including solar-induced fluorescence of vegetation.

[0008] However, both LESS and DART can only do instantaneous simulation and provide only a professional parameter input interface without intuitive real-time model pictures. They are mainly used for remote sensing scientific calculations and have a relatively high threshold for use, so their popularity is still not high. Users are mainly remote sensing scientists, and most people without a remote sensing background find it difficult to understand and use them. At the same time, existing computer simulation methods still have some simplifications for real objects, and the model functions are limited, with fewer simulation parameters and discontinuity in the time dimension. SUMMARY

[0009] Therefore, the present application provides a three-dimensional remote sensing radiation transfer simulation method based on the Omniverse platform, which popularizes the three-dimensional radiation transfer modeling idea through visual radiation transfer process and is more popular, and the obtained data set is more complete.

[0010] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0011] A three-dimensional remote sensing radiation transmission simulation method based on an Omniverse platform is used to simulate a laser radar, comprising the following steps:

[0012] S11, import the pre-established three-dimensional scene model of the test area into the Omniverse platform, and set the ground objects in the three-dimensional scene model as static collision bodies;

[0013] S12, simulate and connect the laser radar and its carrying platform, and set the scanning mode of the laser radar;

[0014] S13, obtain the scanning points of the laser radar through the API interface of the Omniverse platform, combine the collision points of the laser radar with the spatial position of the carrying platform, render the laser radar scan line in real time, and synthesize point cloud data with depth and semantic segmentation information, save the synthesized data and display each frame of information on the window in real time.

[0015] Further, S11 comprises:

[0016] Import the established three-dimensional scene of the test area into the omniverse platform, add a physical simulation module and a physical plane to the virtual scene, set the ground and trees as static collision bodies, and set the collision grid as a convex decomposition algorithm; add semantic segmentation names to selected ground objects through semantic editing models, and automatically add all repeated objects in the scene by adding semantic segmentation names to the source file.

[0017] Further, S12 comprises:

[0018] Create a carrying platform: create four cylinders as wheels and a cube as a car body, set the attribute x = -90° to place the cylinder horizontally; add a rotating joint to the wheels through the stage interface and give a rotation angle transmission, and set the appropriate speed in the joint attribute interface; link the wheels and the car body as a whole to get a wheeled robot, and place it on the ground;

[0019] Create a backpack-type laser radar: enable the line rendering tool, simulate an n-line laser radar, and set the horizontal frequency, frame rate and sampling frequency; connect the created laser radar to the carrying platform, set the scanning mode to rotating scanning, and then connect the laser radar to the carrying platform

[0020] Create an airborne laser radar: select the wheeled robot and place it in an invisible virtual plane in the air, control the wheeled robot to move in the vertical direction to simulate a UAV; set the rotation around the x-axis and the rotation around the z-axis in the rotation attribute of the airborne laser radar to -90°, limit the horizontal field of view angle to 60°, and change the scanning mode to z-scan mode.

[0021] Further, S13 comprises:

[0022] The scan point information of the laser radar is obtained through the API interface of the Omniverse platform, and the scan point information is in the XYZ format of a 3-dimensional array;

[0023] The depth information and semantic segmentation information of each scan point are read through the API interface and added to the corresponding original scan point information;

[0024] The spatial position of the carrying platform corresponding to each frame of scan point is obtained through the API interface, and the position information is added to the corresponding XYZ. The XYZ of the laser radar scan point is converted to the same spatial coordinate system as the carrying platform through a three-dimensional point cloud rotation method;

[0025] All information is output as a txt file, and each line saves the XYZ, depth and semantic segmentation id of a scan point. The synthesized information of each frame is output to a custom UI window for real-time display.

[0026] Further, the corresponding combination of the laser radar scan point and the spatial position of the carrying platform is divided into two cases:

[0027] Case one: when the scanning method of the laser radar is rotary and there is no modified transform attribute, the coordinate system of the laser radar and the carrying platform is the same, and the point cloud information is added according to the corresponding relationship of the coordinate axes; the transform attribute includes: rotate, scale and translate attributes;

[0028] Case two: when the scanning method of the laser radar is z-scan or the transform attribute is changed, the corresponding relationship of the laser radar and the carrying platform spatial information changes, and the data is corrected through a translation and rotation transformation formula;

[0029] The corrected scan point information and the world coordinate system of the carrying platform are added to obtain the final simulated point cloud data.

[0030] Further, the correction method in case two comprises:

[0031] By modifying the rotate attribute of the laser radar, the scanning method is changed to z-scan, the laser radar is first rotated by a certain angle around the x-axis, then rotated by a certain angle around the y-axis, and then the horizontal field of view angle is limited within a predetermined angle range, and the correction formula is:

[0032]

[0033] wherein XYZ represents the point cloud coordinates before correction, X'Y'Z' represents the point cloud coordinates after correction, a x represents the angle of rotation around the x-axis, ay denotes the angle of rotation around the y-axis;

[0034] If the scale property and the translate property are modified, the lidar scan point information follows the same changes, and the correction formula is:

[0035]

[0036] where S x , S y , S z denotes the value of the scale property, T x , T y , T z denotes the value of the translate property.

[0037] Further, the bidirectional reflectance factor is simulated, including the following steps:

[0038] S21, collect the reflectivity information of real ground objects in the test area, and assign the reflectivity information of the real ground objects to the corresponding virtual ground objects in the three-dimensional scene model through the shader;

[0039] S22, calculate the continuous change of the solar azimuth and elevation angles in a day according to the latitude and longitude information and time of the test area, and set the light source distance and light source radius;

[0040] S23, place a virtual target near the target to be observed, simulate the target image through the RTX rendering pipeline, and generate a linear regression model of reflectivity and average gray value;

[0041] S24, set a virtual camera that is fixed at the observation center and moves in a semicircle around the scene center, obtain the multi-angle image of the ground object, extract the average gray value of the pixels within a fixed solid angle in each image, calculate the corresponding reflectivity through the linear regression model, and calculate the continuous change of the bidirectional reflectance factor in a day along with the change of the solar azimuth and elevation angles.

[0042] Further, the shader in S21 uses a layered blending model to adjust the linear combination weight to simulate the reflection and transmission physical effects, wherein the layered blending model is a linear combination of the bidirectional reflectance distribution function and the bidirectional transmission distribution function;

[0043] The calculation formula of the ground surface reflectivity p r is:

[0044] p r =(1-subsurface)×base×base coloor

[0045] The calculation formula of the ground surface transmittance p tThe calculation formula is:

[0046] p t = subsurface x subsurface color

[0047] Wherein, subsurface represents the linear weight of bidirectional transmission distribution function, base represents diffuse reflection weight, base color represents diffuse reflection color, subsurface color represents transmission color.

[0048] Further, S22 comprises:

[0049] Taking the central longitude and latitude of the test area as the origin of the virtual scene, the change of the sun's altitude and azimuth is obtained every day;

[0050] The continuous change of the sun's position is input into the Replicator module of the omniverse platform as the basis for subsequent calculation of the bidirectional reflectance factor in a day; the light source of the scene is set as a disc lamp, which approximates to simulate parallel light incidence, and the radius of the light source is the same as the radius of the circumscribed circle of the scene, covering the entire scene.

[0051] Further, S23 comprises:

[0052] Three virtual gray scale targets are set, with reflectivity of 5%, 10% and 20% respectively, and semantic segmentation names are added for subsequent image segmentation;

[0053] The gray scale targets are placed near the virtual target, the rendering pipeline is switched to RTX, the tone mapping of the rendering pipeline is changed to linear, and the change from sRGB to Gamma space is cancelled, so that the output image is a linear space result;

[0054] The original output format of the Replicator module is modified, and the output content is converted into a one-dimensional array;

[0055] A orthographic image containing all the targets is rendered through the Replicator module, and the bounding box file of each target is output, the bounding box file is read, and the vertex data corresponding to each target is obtained;

[0056] The three targets are segmented into different images using the Opencv library of python, and the average gray value of each target image is calculated, the virtual target and the reflectivity are linearly regressed with the average gray value, and the linear regression equation of the reflectivity and the average gray value is obtained.

[0057] Via the technical solutions, compared with the prior art, the application can simulate the laser radar and the bidirectional reflection factor, and has the following beneficial effects in the simulation of the laser radar:

[0058] 1、The laser radar scanning process of the application is real-time visual output, not only the movement process of the carrying platform, but also the rendering of the laser radar scanning line, so that the user can intuitively see the entire scanning process of the backpack or airborne laser radar, and the real-time output of data can show the generation process of the point cloud data, so that beginners can have a perceptual understanding of the entire process.

[0059] 2、The carrying platform of the application has a physical simulation function, and when the carrying platform encounters obstacles, terrain and different ground, it will have the same effect as the real world, and at the same time, a complex route can be set for the carrying platform to complete a route that is difficult to realize in reality.

[0060] 3、The scanning method and scanning frequency of the virtual laser radar are not limited, and almost all types of laser radars can be simulated, and the feasibility of changing the scanning method and scanning frequency can also be tested to guide the laser radar manufacturers to select the best scanning frequency.

[0061] 4、Compared with the real-world point cloud data acquisition, which requires expensive equipment and a large amount of manpower, it is difficult to obtain data under the forest in a complex forest, and multiple repeated measurements are required to obtain better data results, etc. The virtual laser radar not only has low cost but also can simulate any scene, and virtual data is easy to obtain.

[0062] 5、Since the segmentation of real-world point cloud data requires manual marking, the workload is large, the efficiency is low, and the segmentation accuracy is low, the application can carry segmentation data through simulated point cloud, without the need for manual marking, and the angle and efficiency are very high.

[0063] In the simulation of the bidirectional reflection factor, the following beneficial effects are achieved:

[0064] 1、The optical virtual camera simulation has the advantages of low cost, real-time visualization, physical simulation, virtual scene observation, etc. At the same time, the optical simulation can output more information than the real optical virtual camera, such as the depth information of the object distance from the camera focus or the camera plane distance, the bounding box information containing the minimum rectangular frame of the visible part or the whole of the object, the relative motion information between the object containing pixels and the reframe, etc.

[0065] 2、The application can synthesize a large amount of training data according to the corresponding requirements through the approximation of the physical world, and improve the efficiency of deep learning.

[0066] 3. During optical data measurement, rain, cloudy weather, and strong winds can interfere with the operation of the platform, often causing measurements to be paused and data to become discontinuous. Furthermore, traditional methods cannot acquire data from rainy, cloudy, or windy conditions. However, the virtual platform described in this application can operate normally regardless of interference, acquiring continuous and difficult-to-observe data. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0068] Figure 1 A flowchart for simulating lidar provided by the present invention;

[0069] Figure 2 A flowchart for simulating the biaxial reflectance factor provided by the present invention;

[0070] Figure 3 This invention provides a visual mobile simulation interface for a backpack-type lidar.

[0071] Figure 4 The image shows a visual simulation of the backpack-type lidar provided by this invention.

[0072] Figure 5 This invention provides a visual mobile simulation interface for airborne lidar.

[0073] Figure 6 This is a schematic diagram illustrating the visual motion simulation effect of the airborne lidar provided by the present invention. Detailed Implementation

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] This invention discloses a three-dimensional remote sensing radiative transfer simulation method based on the Omniverse platform, which consists of two parts: simulating lidar and simulating the biaxial reflectivity factor (BRF).

[0076] Among them, such as Figure 1 As shown, the steps for simulating a lidar include:

[0077] S11, import the pre-established three-dimensional scene model of the test area into the Omniverse platform, and set the ground features in the three-dimensional scene model as static collision bodies;

[0078] S12, simulate and connect the laser radar and its carrying platform, and set the scanning mode of the laser radar;

[0079] S13, obtain the scanning points of the laser radar through the API interface of the Omniverse platform, combine the collision points of the laser radar with the spatial position of the carrying platform, render the laser radar scanning line in real time, synthesize the point cloud data with depth and semantic segmentation information, save the synthesized data, and display each frame of information in real time on the window.

[0080] As shown in Figure 2 , the step of simulating the bidirectional reflectance factor includes:

[0081] S21, collect the reflectivity information of the real ground features in the test area, and assign the reflectivity information of the real ground features to the corresponding virtual ground features in the three-dimensional scene model through the shader;

[0082] S22, calculate the continuous change of the solar azimuth and elevation angles in a day according to the latitude and longitude information and time of the test area, and set the light source distance and light source radius;

[0083] S23, place a virtual target near the target to be observed, simulate the target image through the RTX rendering pipeline, and generate a linear regression model of reflectivity and average gray value;

[0084] S24, set a virtual camera that is unchanged at an observation center and performs half-circle motion around the scene center, obtain multi-angle images of the ground features, extract the average gray value of the pixels within a fixed solid angle in each image, calculate the corresponding reflectivity through the linear regression model, and calculate the continuous change of the bidirectional reflectance factor in a day with the change of the solar azimuth and elevation angles.

[0085] Next, each step will be further described.

[0086] In terms of laser radar simulation, specifically:

[0087] S11, the three-dimensional scene of the established test area is imported into the omniverse platform, a physical simulation module and a physical plane are added to the virtual scene, the ground and trees are set as static collision bodies, and the collision grid is set as a convex decomposition algorithm; in Isaac sim (a robot simulation tool platform), a Replicator module is used, semantics segmentation names are added to selected ground objects through a Semantics Schema Editor, and at the same time, the omniverse platform has a built-in function module for quickly arranging a scene, for redundant and repetitive ground objects, semantics segmentation names are added to source files, so that all repetitive objects in the scene are automatically added.

[0088] S12, the simulation of the backpack type laser radar includes:

[0089] Creating a mounting platform: four cylinders are created as wheels, a cube is created as a vehicle body, and the attribute x = -90° is set to place the cylinder horizontally; for the four wheels, a rotating joint is added to the wheel through a stage interface and a rotation angle transmission is given, and a suitable speed is set in the joint attribute interface; the wheels and the vehicle body are linked as a whole to have higher simulation degree, and finally a wheeled robot is obtained and placed on the ground.

[0090] Creating a backpack type laser radar: a laser radar LiDAR is created in Isaac sim, the drawlines are checked to enable line rendering tools, a 40-line laser radar is simulated, the horizontal frequency and frame rate are set, the sampling frequency is controlled at 5000 Hz, the created laser radar is connected to the mounting platform, the scanning mode is set to rotary scanning, and then the laser radar is connected to the mounting platform. As shown in Figure 3 , it is a visual moving simulation interface of the backpack type laser radar.

[0091] For the simulation of the recording laser radar, it includes:

[0092] Creating an airborne laser radar: the wheeled robot is created in the same way as described above, except that the wheeled robot is not placed on the ground, but on an invisible virtual plane in the air, and the wheeled robot is controlled to move in the vertical direction to simulate a UAV; the rotation around the x-axis and the rotation around the z-axis in the rotation attribute of the airborne laser radar are both set to -90°, the horizontal field of view is limited to 60°, and the scanning mode is changed to z-scan mode. As shown in Figure 4 , it is a visual moving simulation interface of the recording laser radar.

[0093] S13, obtain the scanning point information of the laser radar through the API interface of the Omniverse platform, the original scanning point information is a 3-dimensional array in NPY format, and the 3-dimensional array needs to be converted into XYZ format;

[0094] The depth information and semantic segmentation information of each scanning point are read through the API interface, and this information is added to the corresponding original scanning point information;

[0095] The spatial position of the carrying platform corresponding to each frame of laser radar scanning point is obtained through the API interface, and the position information is added to the corresponding XYZ. Since the laser radar LiDAR and the carrying platform use different spatial coordinate systems, the XYZ of the laser radar scanning point needs to be converted to the same spatial coordinate system as the carrying platform through a three-dimensional point cloud rotation method;

[0096] All information is output as a txt file, each line saves the XYZ, depth and semantic segmentation id of a scanning point, and the synthesized information of each frame is output to a custom UI window for real-time display.

[0097] Among them, the combination of scanning point and carrying platform spatial position information is that the virtual world coordinate system is used in the omniverse, and the LiDAR uses a coordinate system with the carrying platform as the origin. Therefore, the corresponding combination of LiDAR scanning point and its carrying platform spatial position is divided into two cases:

[0098] Case one: when the scanning method of the laser radar is rotary and the transform attribute is not modified, the coordinate system of the laser radar and its carrying platform is the same, and the point cloud information is added according to the coordinate axis correspondence; The transform attribute includes: rotate, scale and translate attribute.

[0099] Case two: when the scanning method of the laser radar is z-scan or the transform attribute is changed, the corresponding relationship of the laser radar and its carrying platform spatial information changes, and the data needs to be modified and then combined. The data is modified through a translation and rotation transformation formula; The specific modification method includes:

[0100] 1) Modify the rotate attribute of the laser radar to change the scanning method to z-scan, rotate the laser radar around the x-axis by a certain angle, then rotate it around the y-axis by a certain angle, and then limit the horizontal field of view angle to a preset angle (30 degrees in this embodiment), and the modification formula (right-handed coordinate system, rotation in XYZ order) is:

[0101]

[0102] Wherein, XYZ represents the point cloud coordinates before correction, X'Y'Z' represents the point cloud coordinates after correction, alpha x represents the angle of rotation around the x-axis, alpha y represents the angle of rotation around the y-axis; the angle of the embodiment of the application is fixed at 90°, so it can be simplified as:

[0103]

[0104] 2) If the scale attribute and the translate attribute are modified, the LiDAR scan point information will change in the same way as the two, which will cause the result to be deformed, and the scan point information needs to be corrected, and the correction formula is:

[0105]

[0106] Wherein, S x , S y , S z represents the value of the scale attribute, T x , T y , T z represents the value of the translate attribute.

[0107] It should be noted that both the LiDAR and the mounting platform have transform attributes, and modifying any of them will affect the result, usually modifying the scale attribute of the mounting platform to change the size to adapt to different scenes, and modifying the rotate and translate attributes of the LiDAR to change the scanning method.

[0108] Finally, the corrected scan point information X'Y'Z' is added to the world coordinate system of the mounting platform to obtain the final simulated point cloud data.

[0109] In terms of bidirectional reflectance factor simulation, specifically including:

[0110] S21, using a drone carrying a RedEdge lens to collect forest multispectral data in the test area, then converting the multispectral data into reflectance information through mateshape software, importing the three-dimensional scene model of the test area into the omniverse platform, assigning the reflectance information of the real objects to the corresponding virtual objects through the OmniSurface shader provided by the omniverse, and completing the setting of the virtual forest through fast scene arrangement.

[0111] Among them, the reflectance information of the real objects is assigned to the corresponding virtual objects through the OmniSurface shader provided by the omniverse, specifically including:

[0112] The reconstructed digital twin model (three-dimensional scene model) is input into the omniverse platform, and the ground object reflectivity is calculated according to the following rendering equation.

[0113] L o (q,ω o )=L e (q,ω o )+∫ Ω+ f(q,ω i ,ω o )L i (q,ω i )|cosθ i |dω i (4)

[0114] In the formula, L o (q,ω o ) is the radiance emitted from the point q in the scene along the direction ω o ; L e (q,ω o ) is the radiance emitted by the point along the direction ω o (for example, thermal radiation); f(q,ω i ,ω o ) represents the bidirectional reflectance distribution function (Bidirectional Reflectance Distribution Function, BRDF) near the point; L i (q,ω i ) is the incident radiance near the point along the direction ω i ; θ i represents the angle between the incident light and the local surface normal vector. The entire integral formula represents that the radiance emitted from the point along the direction ω o is equal to the sum of the radiance emitted by the object itself at the point and the radiance scattered by all incident radiance near the point to the outgoing direction.

[0115] In particular, when the incident surface element is a Lambertian body and does not emit light and does not absorb energy itself, the following can be obtained:

[0116] L o (q,ω o )=∫ Ω+ f r L i (q,ω i )|cosθ i |dω i =πf r L i (5)

[0117] Then

[0118] Therefore, the BRDF of the Lambertian body is a constant, and when there is no absorption of energy and total reflection occurs, the value of the BRDF is fixed as 1 / π, defining a reflectance p, which represents the ratio of the reflected energy of the surface element to the incident energy. The BRDF of the Lambertian body can be expressed as:

[0119]

[0120] According to the above formula, the spectral information is combined with the ray tracing model through the reflectance, but existing renderers usually do not have an input interface for direct reflectance, including the omniverse platform used by the present application, and instead input the reflectance into the renderer through a shader. The omniSurface shader in the omniverse platform uses Autodesk Standard Surface, which uses a layered blending model that simulates physical effects such as reflection and transmission by adjusting the linear combination weight. The layered blending model is a linear combination of the bidirectional reflectance distribution function BRDF and the bidirectional transmission distribution function BTDF, and for the BSDF, the following definition is used:

[0121] BSDF=αBRDF+(1-α)BTDF (8)

[0122] In the formula, BSDF represents the bidirectional scattering distribution function, α represents the linear weight of BRDF, and BTDF represents the bidirectional transmission distribution function.

[0123] This model not only consists of reflection and transmission layers, but also includes emission, metal, highlight reflection, back reflection, and other multiple levels. Based on the use purpose of the present application, the specific BSDF of the model is finally expressed as:

[0124] base mix =(1-subsurface)×base×base color ×diffuse brdf(...) +subsurface×subsurface mix (9)

[0125] In the formula, base mix represents BSDF, subsurface represents the weight of BTDF, base is the diffuse reflection weight, base color represents the color of diffuse reflection, and diffuse brdf(...)BRDF represents the Lambertian form.

[0126] subsurface mix =subsurfacec color ×diffuse btdf(...) (10)

[0127] In the formula, subsurface mix Indicates BTDF, subsurface color For transmitted colors, diffuse btdf(...) For the BTDF of the transmissive Lambertian body, diffuse transmission is modeled in this standard by an Oren-Nayar microplane BRDF, which is flipped around the color normal to become a BTDF.

[0128] Formula (9) and Formula (10) can be separated into two parts:

[0129] base brdf = (1 - subsurface × base × base) color ×diffuse brdf(...) (11)

[0130] base btdf =subsurface×base color ×diffuse btdf(...) (12)

[0131] It is important to note that because most remote sensing observations are conducted in the nadir direction (with the zenith angle at 0°, i.e., vertically downward), the BRF of observations in this direction is usually referred to as reflectance. Since the Lambertian assumption is used, the BRF is equal in all directions.

[0132] Therefore, base brdf Represents ρ r , base btdf Represents ρ t diffuse brdf(...) and diffuse btdf(...) Let be the energy distribution function in each direction, that is, the magnitude of reflectivity in each direction on a totally reflective surface. Here, because it's a Lambertian surface, the diffuse... brdf(...) diffuse brdf(...) Both are equal to the constant 1.

[0133] Combining the above formulas (7) and (11)-(12), we can obtain the surface reflectance ρ of the ground object. r The calculation formula is:

[0134] ρr = (1 - subsurface) x base x base color (13)

[0135] ρ t The formula for calculating is:

[0136] ρ t = subsurface x subsurface color (14)

[0137] Where subsurface represents the linear weight of the bidirectional transmission distribution function, base represents the diffuse reflection weight, base color represents the diffuse reflection color, subsurface color represents the transmission color.

[0138] The diffuse reflection weight is set to 1 for a Lambertian body, and the diffuse transmission weight is usually set to 0.5, so that the probability of light being reflected and transmitted is 0.5. The reflectivity and transmissivity can be set by base color and subsurface color , both of which have direct input windows in the shader.

[0139] S22, taking the central longitude and latitude of the test area as the origin of the virtual scene, obtaining the change of the solar elevation angle and the azimuth angle every day.

[0140] The calculation of the hour angle and the declination uses the Spencer formula:

[0141]

[0142] δ = -23.45 cos ((360 / 365) (N + 10)) (16)

[0143] The formula for calculating the solar elevation angle and the azimuth angle uses the Michalsky (1988a, 1988b) approximation formula recommended by WMO:

[0144] sin (el) = sin (δ) sin (Φ) + cos (δ) cos (Φ) cos (h) (17)

[0145] sin (az) = -cos (δ) sin (h) / cos (el) (18)

[0146] Where h represents the hour angle, δ represents the declination, Φ represents the latitude, represents the longitude, t represents the mean solar time, el represents the solar elevation angle, az represents the solar azimuth angle, and N represents the accumulated day, with January 1 as 1.

[0147] After that, the continuous solar position change is input to the Replicator module of the omniverse platform as the basis for subsequent calculation of the bidirectional reflectance factor in a day; the light source of the scene is set to a disc lamp far away from the scene to approximate the simulation of parallel light incidence, and the radius of the light source is the same as the radius of the circumscribed circle of the scene, covering the entire scene. The scattered light part uses a dome light, and the rendering setting of the entire scene is set to path tracing-> postprocessing-> tone mapping in the tone mapping operator; it is set to clamp, that is, the radiation value is kept unchanged, and any exposure adjustment is skipped.

[0148] S23, three virtual gray scale targets with reflectivities of 5%, 10% and 20% are set for radiation calibration, and semantic segmentation names are added for subsequent image segmentation;

[0149] The gray scale targets are placed near the virtual target, the RTX (path tracing) rendering pipeline is switched to, the tone mapping of the rendering pipeline is changed to linear, and the change from sRGB to Gamma space is cancelled, keeping the output image as a linear space result; tone mapping to linear and the above clamp are similar operations, and tone mapping is similar to exposure adjustment of the image. The images commonly seen are Gamma space images, which are more consistent with the perception range of the human eye, but not consistent with the real physical quantity. sRGB is a color standard that specifies the color gamut range. All calculation operations in rendering are performed in linear space, which does not conform to human eye perception, but is physically accurate.

[0150] The original output format of the Replicator module is modified because the original basewriter module output cannot be read by python. The original output format is modified from npy to txt file, and the output content is converted to a one-dimensional array;

[0151] A orthographic image containing all the targets is rendered through the Replicator module, and a bounding box file of each target is output. The bounding box file is read to obtain the vertex data corresponding to each target;

[0152] The three targets are segmented into different images using the Opencv library of python, and the average gray value of each target image is calculated. The virtual target and the reflectivity are linearly regressed with the average gray value to obtain the linear regression equation of the reflectivity and the average gray value.

[0153] S24, the multi-angle image simulation of the forest land needs to calculate the position of the virtual camera first, the application has the function of generating a series of position information of the camera in the virtual space by inputting the camera observation center, rotation step and observation range by the user, and integrating this function into the Replicator, so that a virtual camera with an unchanged observation center and a semicircular motion around the center can be directly generated, and then a large number of multi-angle images can be quickly rendered. The rendering result is a three-channel RGB image, which is split into a single-channel image through the opencv library, and the average gray value of the pixels in a fixed solid angle of each image is extracted, and then the corresponding reflectivity is obtained by the regression equation calculated by S23, and the BRF at a certain moment is obtained. According to the above calculation of the solar azimuth angle and the elevation angle, the continuously changing BRF of the whole day is calculated.

[0154] Next, some terms related to the application are explained.

[0155] Quantitative remote sensing: refers to the technology and method of quantitatively extracting surface parameters from the observed electromagnetic wave signal, that is, extracting the actual parameters of the ground from the remote sensing image data, which is the most important application of remote sensing.

[0156] Remote sensing radiation transmission modeling: in order to quantitatively describe the interaction between solar radiation and vegetation canopy, one-dimensional radiation transmission model, geometric optical model, hybrid model and computer simulation model based on three-dimensional structure of vegetation (also known as three-dimensional radiation transmission model) have appeared in turn.

[0157] Remote sensing ill-posed inversion: remote sensing inversion is the reverse process of remote sensing radiation transmission modeling, which is the process of extracting various surface parameters from remote sensing signals. Since the variables provided by remote sensing are usually less than the unknowns, the inversion is essentially an ill-posed inversion problem.

[0158] Prior knowledge (data set): here refers to ground verification data, which can be compared and corrected with the inversion results extracted from remote sensing signals.

[0159] Ray tracing: by tracing the light rays interacting with the optical surface to obtain the path of the light rays, finally, the light rays enter the virtual camera negative, and the picture is generated. Forward ray tracing is to trace the light rays from the light source to the sensor (virtual camera lens) after multiple reflections, backward ray tracing (also known as path tracing) is to trace the light rays from the sensor to the light source, and bidirectional path tracing is to build the path from one end of the light rays from the camera and the other end from the light source, and connect them.

[0160] Triangle patch: a method that can accurately describe the physical real structure, that is, any geometric body in reality can be divided into countless triangle patches, which is the most basic unit of three-dimensional geometric body.

[0161] Omniverse: NVIDIA Omniverse is a graphics and simulation platform from NVIDIA based on NVIDIA RTX and Pixar's Universal Scene Description (USD), with many applications to suit different needs, high degree of modularity and customizable modules.

[0162] LiDAR: LiDAR is a radar system that uses laser beams to detect the position, speed, and other characteristics of targets.

[0163] Reflectance factor: The ratio of the energy reflected in a certain direction on a surface element to the energy in that direction when the surface element is assumed to be an ideal diffuse surface.

[0164] Bidirectional reflectance factor (BRF): BRF is the most measurable physical quantity closest to BRDF, which is defined as the ratio of the energy reflected by a target in a certain observation direction to the energy reflected when the target is replaced by an ideal diffuse surface under direct light incidence. The continuous distribution of the reflectance factor in the hemisphere is the bidirectional reflectance factor.

[0165] Relationship between reflectivity and reflectance factor: Reflectivity and BRDF are difficult to measure in reality, so under the assumption of uniform and Lambertian surface of the observed target, reflectance factor and BRF, which are easy to measure, are used instead of reflectivity and BRDF.

[0166] Radiometric calibration: Radiometric calibration is the process of converting the voltage signals or digital values (DN) recorded by the sensor into absolute dimensional radiance or reflectivity.

[0167] Isaac sim: an application in Omniverse, containing modules related to LiDAR and simulated robots.

[0168] Replicator module: Omniverse Replicator is a highly scalable framework built on the scalable Omniverse platform, which can generate physically accurate 3D synthetic data to accelerate the training and performance of AI perception networks.

[0169] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0170] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the appended claims are intended to cover all such modifications that do not depart from the true spirit and scope of the application. Therefore, the application is not limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A three-dimensional remote sensing radiative transfer simulation method based on the Omniverse platform, characterized in that, The steps for simulating lidar include: S11. Import the pre-established 3D scene model of the test area into the Omniverse platform, and set the ground features in the 3D scene model as static collision bodies; S12. Simulate and connect the lidar and its mounting platform, and set the lidar's scanning mode; S12 includes: Create a platform: Create four cylinders as wheels and a cube as the body. Set the x-axis to -90° to place the cylinders horizontally. Add rotational joints to the four wheels through the stage interface and assign them angular transmission. Set the appropriate speed in the joint properties interface. Add links to the wheels and the body to obtain a wheeled robot and place it on the ground. Create a backpack-style LiDAR: Enable the line rendering tool to simulate an n-line LiDAR, and set the horizontal frequency, frame rate, and sampling frequency. Connect the created LiDAR to the mounting platform, set the scanning mode to rotational scanning, and then connect the LiDAR to the mounting platform. Create an airborne LiDAR: Select a wheeled robot and place it in an invisible virtual plane in the air. Control the wheeled robot to move vertically to simulate a drone. Set the rotation properties of the airborne LiDAR to -90° around the x-axis and around the z-axis, limit the horizontal field of view to 60°, and change the scanning mode to a zigzag scanning mode. S13. Obtain the scanning points of the LiDAR through the API interface of the Omniverse platform, combine the collision points of the LiDAR with the spatial position of the platform, render the LiDAR scanning lines in real time, and synthesize point cloud data with depth and semantic segmentation information. Save the synthesized data and display the information of each frame in real time on the window. This method is also used to simulate the bidirectional reflectance factor, including the following steps: S21. Collect reflectance information of real objects in the test area, and assign the reflectance information of real objects to the corresponding virtual objects in the three-dimensional scene model through a shader; S22. Calculate the continuous changes in solar azimuth and altitude angles throughout the day based on the latitude and longitude information and time of the test area, and set the distance and radius of the light source. S23. Place a virtual target near the target to be observed, simulate the target image through the RTX rendering pipeline, and generate a linear regression model of reflectivity and average gray value. S24. Set up a virtual camera with a fixed observation center that moves in a semi-circle around the center of the scene to acquire multi-angle images of ground features. Extract the average gray value of pixels within a fixed solid angle of each image and calculate the corresponding reflectance using a linear regression model. Calculate the continuous change of the bidirectional reflectance factor within a day as the solar azimuth and altitude angles change.

2. The three-dimensional remote sensing radiative transfer simulation method based on the Omniverse platform according to claim 1, characterized in that, S11 includes: Import the established 3D scene of the test area into the Omniverse platform, add a physical simulation module and physical plane to the virtual scene, set the ground and trees as static colliders, and set the collision mesh to convex decomposition algorithm; add semantic segmentation names to the selected features through semantic editing model, and add semantic segmentation names to the source file for redundant and duplicate features so that all duplicate objects in the scene are automatically added.

3. The three-dimensional remote sensing radiative transfer simulation method based on the Omniverse platform according to claim 1, characterized in that, S13 includes: The scanning point information of the LiDAR is obtained through the API interface of the Omniverse platform. The scanning point information is a 3D array in XYZ format. The depth and semantic segmentation information of each scan point are read through the API interface and added to the corresponding original scan point information; The spatial position of the mounting platform corresponding to each frame scan point is obtained through the API interface, and its position information is added to the corresponding XYZ. The XYZ of the LiDAR scan points is transformed to the same spatial coordinate system as the mounting platform through a 3D point cloud rotation method. All information is output as a txt file, with each line storing the XYZ, depth, and semantic segmentation ID of a scan point. The synthesized information of each frame is output to a custom UI window for real-time display.

4. The three-dimensional remote sensing radiative transfer simulation method based on the Omniverse platform according to claim 3, characterized in that, The correspondence between the LiDAR scanning points and the spatial positions of their mounting platforms can be divided into two cases: Scenario 1: When the LiDAR scanning method is rotational and the transform attribute is not modified, the coordinate system of the LiDAR and its mounting platform is the same, and the point cloud information is added according to the coordinate axis correspondence; the transform attribute includes: rotate, scale and translate attributes; Scenario 2: When the LiDAR scanning method is zigzag scanning, or when the transform attribute is changed, the correspondence between the LiDAR and its mounting platform spatial information changes, and the data is corrected by translation and rotation transformation formula; The corrected scan point information is added to the corresponding world coordinate system of the platform to obtain the final simulated point cloud data.

5. The three-dimensional remote sensing radiative transfer simulation method based on the Omniverse platform according to claim 4, characterized in that, The correction methods in Scenario 2 include: By modifying the `rotate` attribute of the LiDAR, the scanning method is changed to a zigzag scan. The LiDAR is first rotated around the x-axis by a certain angle, then around the y-axis by a certain angle, and the horizontal field of view is limited to a preset angle range. The corrected formula is as follows: Where XYZ represents the point cloud coordinates before correction, X′Y′Z′ represents the point cloud coordinates after correction, and α x α represents the angle of rotation about the x-axis. y Indicates the angle of rotation about the y-axis; If the scale and translate attributes are modified, the LiDAR scan point information will change accordingly. The correction formula is as follows: Among them, S x S y S z Represented as the numerical value of the scale attribute, T x T y T z This represents the value of the translate property.

6. The three-dimensional remote sensing radiative transfer simulation method based on the Omniverse platform according to claim 1, characterized in that, The shaders in S21 use a layered blending model to adjust the linear combination weights to simulate reflection and transmission physics effects. The layered blending model is a linear combination of the bidirectional reflection distribution function and the bidirectional transmission distribution function. Surface reflectance ρ r The calculation formula is: r r =(1-subsurface)×base×base color Ground surface transmittance ρ t The calculation formula is: r t =subsurface×subsurface color Where, subsurface represents the linear weight of the bidirectional transmission distribution function, base represents the diffuse reflection weight, and base color Indicates diffuse color, subsurface color Indicates the transmitted color.

7. The three-dimensional remote sensing radiative transfer simulation method based on the Omniverse platform according to claim 1, characterized in that, S22 includes: Using the latitude and longitude of the experimental area as the origin of the virtual scene, the daily changes in solar altitude angle and azimuth angle are obtained; The continuous changes in the sun's position are input into the Replicator module of the omniverse platform as the basis for subsequent calculations of the bidirectional reflectance factor throughout the day; the scene's light source is set as a disc light to approximate parallel light incidence, and the radius of the light source is the same as the radius of the scene's circumcircle, covering the entire scene.

8. The three-dimensional remote sensing radiative transfer simulation method based on the Omniverse platform according to claim 1, characterized in that, S23 includes: Three virtual grayscale targets are set with reflectivities of 5%, 10%, and 20%, respectively, and semantic segmentation names are added for subsequent image segmentation. Place the grayscale target near the virtual target, switch to the RTX rendering pipeline, change the rendering pipeline's tone mapping to linear and cancel the sRGB to Gamma space conversion, keeping the output image a linear space result; Modify the original output format of the Replicator module, transforming the output content into a one-dimensional array; The Replicator module renders an orthophoto containing all targets and outputs the bounding box file for each target. The bounding box file is then read to obtain the vertex data corresponding to each target. Using Python's OpenCV library, three targets are segmented into different images, and the average gray value of each target image is calculated. The virtual target and reflectance are linearly regressed with their average gray values ​​to obtain the linear regression equation between reflectance and average gray values.

Images