A deep learning three-dimensional climate simulation method for adding climate to real scenes

Abstract

The present invention relates to a deep learning three-dimensional climate simulation method for adding climate to a real scene, belonging to the field of image processing, including: inputting a real scene image into the Jittor framework, each pixel of the real scene image emits a ray, sampling each ray to obtain sampling points and directions and encoding them, sending them into a neural radiance field to predict the color value and density value of each sampling point, substituting the parameters predicted by the neural radiance field into the rendering equation and then inputting it into volume rendering, so as to retain the color, depth and opacity of the sampling points; segmenting meteorological disaster elements from the input meteorological disaster pictures and reconstructing the meteorological scene corresponding to the meteorological disaster; fusing the final scenes obtained in the previous two steps of the scene to achieve effect simulation. Through the three-dimensional reconstruction technology, the present invention can intuitively show the impact of meteorological disasters on the environment, enabling users to visually feel the disaster situation, greatly enhancing the data visualization effect and dissemination efficiency.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a deep learning three-dimensional climate simulation method for adding climate to a real scene. Background Art

[0002] The threat of climate change to the earth is becoming more and more serious. With the emergence of global warming, the occurrence frequency of meteorological disaster events has changed, showing an increasing trend. Meteorological disasters such as blizzards, floods, and sandstorms occur more frequently. Research has found that visualizing meteorological disaster situations helps the public more intuitively feel the impact of climate change on a certain scene, thereby increasing the public's attention to meteorological hazards and enhancing the possibility of taking actions to address and solve them. It is also conducive to coming up with more targeted solutions. Imagine that it is difficult to take reasonable measures without seeing the impacts of these weathers.

[0003] However, the current visualization of meteorological disaster scenes is mostly presented in the form of text and meteorological maps, lacking intuitiveness, not being conducive to the public's understanding, and having limited dissemination of meteorological information. Therefore, there is an urgent need for a domestic method that can quickly, efficiently, and accurately perform climate simulation on real scene images to meet the need that in the face of climate disasters, emergency management can provide visualization of a highly realistic disaster-affected environment in a short time and provide a reference for subsequent work. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a deep learning three-dimensional climate simulation method for adding climate to a real scene, solving the deficiencies existing in the prior art.

[0005] The purpose of the present invention is achieved through the following technical solutions: A deep learning three-dimensional climate simulation method for adding climate to a real scene, the simulation method includes:

[0006] Scene reconstruction step: Input a real scene image into the Jittor framework. Each pixel of the real scene image emits a ray to obtain the ray ray(o, d). Sample each ray to obtain the sampling points xyzs and directions dirs. Encode the sampling points xyzs and directions dirs using the instantNGP encoder implemented based on Jittor and send them into the neural radiance field to predict the color value of each sampling point and density value , substitute the parameters predicted by the neural radiance field into the rendering equation and then input it into volume rendering, so as to retain the color, depth, and opacity of the sampling points and achieve scene reconstruction;

[0007] Meteorological simulation step: Input a meteorological disaster picture into the ResUnet framework for meteorological disaster element segmentation to reconstruct the meteorological scene corresponding to the meteorological disaster;

[0008] Scene fusion step: fuse the final scenes obtained from the scene reconstruction step and the meteorological simulation step to obtain a climate simulation map.

[0009] The ray sampling includes:

[0010] Divide the space into a three-dimensional grid composed of a first grid and a second grid. When it is the first sampling step size, use the first grid to capture the main structure of the scene. When it is the second sampling step size, use the second grid for fine sampling to capture the details of the scene, and associate the first grid and the second grid with the grid density. The sampling points are concentrated in the grids with density values greater than the set value to ensure more sampling in areas with higher density;

[0011] Calculate the positional relationship between the picture scene box bounding box and the ray to obtain , , when t min is greater than t max , the ray has no intersection with the bounding box; if t max is less than 0, the ray is behind the bounding box and there is also no intersection. Record the t min and t max when the ray intersects the bounding box. Set the ray starting point as t min , and end the sampling when t is greater than t max or the number of sampling points reaches the preset value. t represents the step parameter of the ray along its path, t min represents the positional relationship between the minimum value of the bounding box in the x-axis direction and the ray, t max represents the positional relationship between the maximum value of the bounding box in the x-axis direction and the ray, min x and max x represent the minimum and maximum values of the bounding box in the x-axis direction respectively, represents the coordinate of the ray starting point on the x-axis, represents the component of the ray direction vector on the x-axis;

[0012] Calculate the size of the sampling step to select the first grid or the second grid, control the sampling points to always be in the area with density value greater than the set value according to the density distribution in the grid, record the t value of the nth sampling point position on each ray into the ts array, and then obtain the three-dimensional coordinates of the sampling points xyzs and the corresponding directions dirs. n represents the number of sampling points, and ts represents the array.

[0013] The meteorological simulation step includes: sandstorm meteorological simulation and flood meteorological simulation;

[0014] The sandstorm meteorological simulation includes: constructing the sandstorm color and sandstorm density according to the Lambert-Beer law based on the depth, opacity, and cumulative effect function, and integrating according to volume rendering to obtain a sandstorm scene color value picture with only sandstorms, and combining with the picture to obtain the sandstorm climate simulation;

[0015] The flood meteorological simulation includes: calculating the water surface mask of the picture and the depth value at the intersection point according to the depth, the center point of the water level, and the normal vector, obtaining the water surface intersection point, simulating the wave points and wave normals required for the water surface waves at the water surface intersection point by the spectral method, calculating the reflection coefficient of these wave points on the water surface through the Fresnel effect, and at the same time sending the obtained reflected rays into ray sampling to obtain the reflected color of the water surface intersection point of the picture, and obtaining the flood climate simulation according to the composition of the water surface mask, ray reflectivity, water body color, and image color.

[0016] Each pixel of the real scene image emits a ray, which includes:

[0017] Using deep learning to learn the three-dimensional scene where the real scene picture is located, and randomly obtaining the color value rgb_tar of the pixel point and the pixel position xy in the pixel coordinate system from the rgb_index of each picture according to the batch size, where rgb_index represents the picture and rgb_tar represents the color value;

[0018] Obtaining ray-related information from the corresponding camera pose matrix T, where the matrix T includes a rotation matrix R and a translation vector trans, and the translation vector trans is the position of the camera;

[0019] Connecting the position coordinates of the camera and a certain pixel on the plane to determine a ray, taking the translation vector trans as the ray origin ray_o, and at the same time calculating the ray direction through the rotation matrix R of the camera, the principal point position p, the resolution res, the focal length f, the pixel position xy, and the translation vector trans Finally, obtaining the ray equation rays(trans)=ray_o+trans·ray_d, where normalize represents vector normalization.

[0020] The specific encoding of the sampling points xyzs and directions dirs using the instantNGP encoder implemented based on jittor includes:

[0021] For the sampling points xyzs, use multi-resolution hash encoding. Encode the three-dimensional space into sixteen resolution levels from low to high. Set a learnable two-dimensional feature for each point at each resolution and store it in the hash table of each resolution. Then, for the given input coordinates xyz, find the eight nearest vertices at each resolution, perform trilinear interpolation on their feature values, and combine the feature values to obtain a 32-dimensional feature vector;

[0022] For the direction dirs, use third-order spherical harmonic function encoding to fit the direction dirs into a 16-dimensional feature vector.

[0023] Feed it into the neural radiance field to predict the color value of each sampling point and density value Specifically, it includes:

[0024] Input the 32-dimensional feature obtained by encoding the sampling points xyzs into the first MLP network to obtain the density value of each sampling point , and at the same time, after encoding the viewing direction and connecting it with the output of the first MLP network, input it into the second MLP network to obtain the predicted color value, and use the activation function Sigmoid for activation to obtain the color value of each sampling point ;

[0025] For each sampling point on each ray, the color value and density value are integrated through volume rendering, and the finally calculated pixel color is used to calculate the loss with the color value rgb_tar of the obtained pixel point to obtain the difference between the real image and the network-generated image.

[0026] The specific steps of step two include:

[0027] By adding a non-negative constant to the density in free space to simulate the sandstorm density in free space, perform operations on the predicted color, depth, and opacity of each obtained pixel point to obtain and ;

[0028] Take the obtained as the distance from each point on each ray to point , Take it as the cumulative effect of the sandstorm, take the sandstorm color and sandstorm density as the output of the neural radiance field, and let volume rendering integrate according to to obtain the scene color value C of only the sandstorm dust ;

[0029] The obtained scene color value C dust Perform a shallow sampling through the neural radiance field, and use C dust As the output, perform volume rendering on the original color again to obtain the sandstorm climate simulation scene.

[0030] The specific steps of calculating the water surface mask and the depth value at the intersection point according to the center point of the depth and the horizontal plane and the normal vector, and obtaining the water surface intersection point include:

[0031] In three-dimensional space, use the point-normal form to represent a plane, that is, a three-dimensional coordinate , and the spatial normal vector normal=(a, b, c), to obtain the plane equation , where a, b, and c are the three components of the normal vector of the plane, representing the components of the plane normal vector on the x-axis, y-axis, and z-axis respectively;

[0032] Through the formula Calculate the depth value dep from the intersection point of the ray and the plane to the camera pose, and compare it with the predicted depth depth of each pixel point. When dep is greater than 0, it indicates that the ray intersects the plane. When dep is less than 0, it indicates that the ray has no valid intersection with the plane. When dep is 0, the ray origin is on the plane;

[0033] When dep is less than 0, set the ray depth to the maximum value in depth. By judging whether dep is less than depth, determine whether there is a closer intersection point caused by other objects or surfaces before each ray intersects the horizontal plane, obtain a water surface mask is_water, and perform visualization to obtain a black and white picture;

[0034] Retain the data where the ray first intersects the water surface through the water surface mask is_water.

[0035] The specific steps of simulating the wave points and wave normals required for the water surface waves at the water surface intersection point by the spectral method include:

[0036] Through the formula Calculate the intersection point coordinates of the ray that first intersects the water surface on the water surface, which is the intersection point of the ray and the water surface;

[0037] Through Use the inverse fast Fourier transform to convert the frequency domain information to the spatial domain, obtain the actual shape and normal vector of the wave, and get the wave normal ;

[0038] Through the formula Calculate the reflection ray direction of the wave water surface intersection point, and Expand it into a matrix with the same number of rays as the ray, denoted as , where, Represents the ellipsis in the matrix, i.e., the repeated elements in the matrix , the dimension of matrix N’ is , the parameter O represents the real number field, that is, the elements in the matrix are real numbers, and u represents the number of rows of the matrix;

[0039] Construct orthogonal basis vectors, calculate the scattering directions to obtain the main direction and four scattering directions, and combine them into a tensor to obtain the scattering direction as , and unit vectorize each scattering direction to , where L represents the original incident ray direction, Lx0, Lx1 and Ly0, Ly1 respectively represent the four scattering directions under the action of water surface waves, and they are the scattering components generated by the interaction between the wave surface and the incident ray;

[0040] Calculate the weight W and perform weight normalization so that the sum of the weights of each ray is 1, record the obtained scattering direction S and the weights W of each direction, and use the scattering direction S and as the direction and starting point of the ray to obtain the pixel value C_ref of the water surface reflection and scattering;

[0041] Superimpose these scattering rays representing the waves according to the weights obtained from each ray to obtain the water surface reflection rgb_reflect simulating the wave effect.

[0042] Calculating the reflection coefficient of these wave points on the water surface through the Fresnel effect, and at the same time sending the obtained reflected rays into ray sampling to obtain the reflected color at the intersection of the picture horizontal plane specifically includes:

[0043] Calculate the reflection coefficient of the water surface according to the Fresnel effect , , , where is the reflection coefficient of vertically polarized light, is the reflection coefficient of horizontally polarized light, is the refraction angle, is the incident angle;

[0044] Sum the dot products of the reflected ray directions and to obtain the incident angle θ, and calculate the reflectivity ;

[0045] According to the formula calculate the water surface color , is the preset water body color;

[0046] Fuse the water surface mask with the original scene color C to obtain the flood climate simulation.

[0047] The present invention has the following advantages: A deep learning three-dimensional climate simulation method for adding climate to real scenes realizes the three-dimensional dynamic visualization of meteorological disaster scenes. By introducing high-quality three-dimensional rendering technology, users can more intuitively feel the impact of meteorological disasters (such as floods, blizzards, sandstorms) on the environment. This intuitive display method not only helps improve the public's understanding of meteorological disasters but also enhances the dissemination effect of meteorological information, while meeting the needs of various aspects such as emergency management, meteorological science popularization, urban planning, environmental protection, and energy security; by setting thresholds in advance, when a certain element in the input meteorological image reaches the corresponding threshold, the relevant effects of the scene environment are automatically changed. For example, when the reservoir water level reaches the flood control high water level, the sky automatically becomes dim, indicating the arrival of a flood disaster; it can generate high-precision meteorological scenes more quickly, significantly improving the authenticity and diversity of the scenes, and providing high-precision simulation support for disaster scenes such as floods and blizzards; it supports dynamic simulations of multiple meteorological styles, including floods, sandstorms, blizzards, etc. The generated scenes are more natural and realistic in visual effects and can be flexibly applied to different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow diagram of the present invention;

[0049] Figure 2 It is a schematic diagram of the scene color value of the scene with only sandstorms;

[0050] Figure 3 It is a schematic diagram of the sandstorm climate simulation scene after re-performing volume rendering;

[0051] Figure 4 It is a schematic diagram of the dep visualization in the flood climate simulation;

[0052] Figure 5 It is a schematic diagram of the depth of each pixel predicted by the network model in the flood climate simulation;

[0053] Figure 6 It is a schematic diagram of the black and white picture after the water surface mask is visualized in the flood climate simulation;

[0054] Figure 7 It is a schematic diagram of the water surface reflection with the simulated wave effect superimposed in the flood climate simulation;

[0055] Figure 8 It is a schematic diagram of the water surface color calculated in the flood climate simulation;

[0056] Figure 9 It is a schematic diagram of the final flood climate simulation scene obtained by fusion in the flood climate simulation. DETAILED DESCRIPTION OF THE INVENTION

[0057] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Usually, the components of the embodiments of this application described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the protection scope of the claimed application, but only represents the selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative efforts belong to the protection scope of this application. The following further describes the present invention with reference to the accompanying drawings.

[0058] It should be noted that the explanations of some technical terms involved in the present invention are as follows:

[0059] Jittor: A domestic deep learning framework based on a Just In Time (JIT) compiler. By compiling code with a high repetition rate into machine code and creating a high-performance custom operator in the form of code, etc., it realizes efficient computing performance and greatly shortens the reconstruction time.

[0060] instantNGP: An efficient neurographics technology developed by the NVIDIA research team, aiming to significantly accelerate the training and inference processes of neural rendering and related tasks.

[0061] Spherical harmonic function: An effective tool for representing direction-dependent quantities in three-dimensional space and has wide applications in computer graphics.

[0062] MLP: Full name is Multilayer Perceptron, a multi-layer perceptron, which is a basic form of artificial neural network. An MLP consists of an input layer, one or more hidden layers, and an output layer. The layers are connected by weights, but there are no connections within the layer, forming a feedforward network structure, that is, the signal can only flow from the previous layer to the next layer and cannot form a loop.

[0063] As Figure 1 shown, the present invention specifically relates to a domestic jittor deep learning three-dimensional climate simulation method for adding sand dust and flood climates to a real scene, which specifically includes three parts: three-dimensional scene reconstruction, meteorological simulation, and scene fusion.

[0064] Furthermore, the 3D scene reconstruction includes: transplanting the part of 3D scene learning from PyTorch to a domestic deep learning framework, reconstructing based on NeRF using the Jittor framework with real scene images input, and improving ray sampling. That is, a ray is emitted from each pixel of the real scene image to obtain the ray ray(o,d), where o represents the origin of the ray and d represents the direction of the ray. Then, each ray is sampled to obtain the sampled points xyzs and directions dirs. Then, the sampled points xyzs and dirs are encoded using the instantNGP encoder implemented based on Jittor and fed into the neural radiance field to predict the color value of each sampled point and density value ; Finally, the parameters predicted by the neural radiance field are substituted into the rendering equation and then input into volume rendering, so as to render the color, density of the sampled points into the color, depth and opacity of the pixels and retain them.

[0065] Among them, the 3D scene reconstruction specifically includes the following contents:

[0066] 1. Multi-view image input and data collection;

[0067] Collect multi-view images, and use clomap to obtain the camera focal length, camera pose, and image principal point. Preprocess the images by normalizing them to ensure the consistency of the input data.

[0068] 2. Obtain rays;

[0069] First, use deep learning to learn the 3D scene where the real scene image is located. According to the batch size, randomly obtain the color value (rgb_tar) of the pixel points and the pixel position (xy) in the pixel coordinate system from each image (rgb_index), and obtain the ray-related information from the corresponding camera pose matrix T. In this 4×4 matrix, the upper left 3×3 matrix R represents rotation; the first three rows in the fourth column form a vector represents translation, that is, the position of the camera (world coordinate system).

[0070] ,

[0071] Connect the position coordinates of the camera and a certain pixel on the plane to determine a ray. Take out the vector trans and use it as the ray origin ray_o (triple). At the same time, through the camera rotation matrix R, the principal point position p, the resolution res, the focal length f, the pixel position xy, and the translation vector trans, calculate the ray direction ray_d (three-dimensional vector). The ray direction is:

[0072] ,

[0073] Among them, normalize represents vector normalization, that is, turning the vector into a unit vector, so that the ray equation can be obtained: .

[0074] 3. Ray sampling;

[0075] Introduce a sampling strategy: Since computer resources are limited and it is impossible to record all the information of each point on the ray, an effective sampling strategy is needed to improve the sampling accuracy and reduce the time consumed by invalid sampling. Therefore, a coarse-to-fine sampling strategy is adopted, which can balance the computing resources and sampling accuracy.

[0076] Space division and grid selection: Divide the sampling space into three-dimensional grids from coarse to fine. When the sampling step size is large, use a coarse grid to capture the main structure of the scene; when the sampling step size is small, use a fine grid for fine sampling to capture the details of the scene. Here, the large and small step sizes are relative. As long as there is a large and a small step size, the large one is the large step size and the small one is the small step size. The same applies to the coarse and fine grids.

[0077] Grid density and sampling points: Associate the coarse and fine grids with the grid density. Sampling points will be concentrated in the grids with larger density values to ensure more sampling in higher-density areas and avoid wasting resources.

[0078] Calculate the intersection point of the ray and the bounding box: Calculate the positional relationship between the bounding box of the picture scene frame and the ray, , , where min x and max x respectively represent the minimum and maximum values of the bounding box in the x-axis direction, represents the x-axis coordinate of the starting point of the ray, represents the x-axis component of the direction vector of the ray. When t min is greater than t max , the ray has no intersection with the bounding box; if t max is less than 0, the ray is behind the bounding box and there is also no intersection; record t min and t max (representing the intersection point of the ray and the bounding box), then set the starting point of the ray to t min , and perform stepping through . When t is greater than t maxEnd sampling when the sampling point reaches the preset value, then calculate the sampling step size to select a coarse grid or a fine grid, and control the sampling point to always be in the region with a larger density (hitting the object surface) according to the density distribution in the grid, and record the t value of the nth sampling point position on each ray into the ts array. In this way, the three-dimensional coordinates xyzs of the sampling points and the corresponding viewing directions dirs are obtained.

[0079] 4. Encode xyzs and dirs;

[0080] If the obtained three-dimensional coordinates xyzs of the sampling points and the corresponding viewing directions dirs are directly put into the deep learning network, high-frequency information cannot be learned, and the learnable features are insufficient (the dimension is only 3). It is necessary to map these low-dimensional data from low to high to a multi-dimensional space through an encoder so that the MLP can learn sufficient features. At the same time, efficient encoding can use a smaller network to learn sufficient details.

[0081] For the information expressed by xyzs and dirs, different encodings are selected for processing. Multiresolution hash encoding is used for xyzs. The three-dimensional space is encoded according to sixteen resolution levels from low to high. A learnable two-dimensional feature is set for each point at each resolution and stored in the hash table of each resolution. Then, for the given input coordinate xyz, find the nearest 8 vertices at each resolution, perform trilinear interpolation on their feature values, and then combine the feature values together. Finally, a 32-dimensional feature vector is obtained. In this process, due to the time complexity of the hash function being O, the encoding speed is greatly reduced, and sufficient learning features are obtained; for dirs, the commonly used third-order spherical harmonic function encoding is used to fit dirs into a 16-dimensional feature vector.

[0082] 5. Neural radiance field;

[0083] In the network model, the 32-dimensional feature obtained after encoding xyzs is input into a smaller MLP network to obtain the network output, and the density of each sampling point is obtained , and at the same time, the encoded viewing direction is connected to the output of the smaller MLP network and input into a larger MLP network to obtain the predicted value of the color, which is activated using the activation function Sigmoid to obtain the color of each sampling point .

[0084] At this time, each sampling point on each ray and Through volume rendering, perform integration:

[0085] ,

[0086] where N represents the number of sampling points, represents the distance between adjacent sampling points, represents the transmittance from the i-th sampling point to the end point of the ray. is the color value finally rendered for this ray.

[0087] The finally inferred pixel color , and then calculate the Loss with the previously recorded rgb_tar. Use the mean squared error (MSE) loss function to calculate the difference between the real image and the image generated by the network.

[0088] ,

[0089] where is the real image, is the image generated by the network model.

[0090] 6. Obtaining depth and opacity;

[0091] After learning through the neural radiance field network model, a network model that can obtain a realistic scene simulation photo based on camera-related information can be obtained, that is, an implicit 3D reconstruction model.

[0092] Obtaining pictures of different camera angles of the 3D scene, and the opacity and depth of the scene will also be obtained. Only after the model learns sufficient 3D features of the scene can reliable opacity and depth be obtained, expressed as:

[0093] ,

[0094] .

[0095] Among them, represents the step size between each point sampled along the ray path, represents the density at the j-th sampling point, represents the position of the j-th sampling point.

[0096] Furthermore, the meteorological simulation includes sandstorm meteorological simulation and flood meteorological simulation.

[0097] Among them, the sandstorm climate simulation includes: the depth and opacity information obtained by ResUnet segmentation, the step size between ray sampling points, the cumulative effect function, and according to the Lambert-Beer law (which can describe the relationship between the absorption of light by a substance and the concentration of the substance and the optical path length), constructing the sandstorm color , sandstorm density , Integrating according to volume rendering, the obtained color value image of the sandstorm scene with only the sandstorm is combined with the color value image of the sandstorm scene to obtain the sandstorm climate simulation.

[0098] Furthermore, the sandstorm climate simulation specifically includes the following contents:

[0099] 1. Extract the sand and dust field color;

[0100] Simulate according to Lambert-Beer's law. Lambert-Beer's law can be expressed using volume rendering. Both are integrations of rays. At the same time, the length of each ray and the distance between sampling points are obtained from the model, and the sandstorm color , sandstorm density and cumulative effect .

[0101] By adding a non-negative constant to the density in free space, the sandstorm density in free space is simulated. Perform operations on the opacity and depth in the obtained opacity and depth values:

[0102] ,

[0103] ,

[0104] Among them, represents the step size between each ray sampling point, and the obtained is used as the distance from to point on each ray, represents the cumulative effect of the sandstorm. Taking and as the output of the neural radiance field, let volume rendering integrate according to , and a scene color value C of only the sandstorm as shown in Figure 2 is obtained. dust .

[0105] 2. Resample and synthesize;

[0106] The sandstorm scene color C obtained at this time dust is combined with the C predicted by the network model. According to the original sampling points, C dust is subjected to a shallow sampling according to the neural radiance field steps. Taking C dust as the network output, a volume rendering is performed again on the original C to obtain a sandstorm simulation scene as shown in Figure 3 .

[0107] Furthermore, flood climate simulation includes: calculating the water surface mask of the picture and the depth value at the intersection point to obtain the water surface intersection point by using the depth obtained by ResUnet segmentation, the center point of the water level, and the normal vector; simulating the wave points and wave normals required for the water surface waves at the water surface intersection point by using the spectral method; calculating the reflection coefficient of these wave points on the water surface through the Fresnel effect (which describes the reflection and refraction behavior of rays at the interface of different media); meanwhile, sending the obtained reflected rays into ray sampling, and sequentially executing to obtain the reflected color of the water level intersection point in the picture; and obtaining the flood climate simulation according to the water surface mask, ray reflectivity, water body color, and image color combination.

[0108] Furthermore, the flood climate simulation specifically includes the following:

[0109] When simulating floods, it is necessary to determine a water level plane for the entire three-dimensional scene and judge the position submerged by the water surface. There are many ways to represent a plane in three-dimensional space, and the present invention uses the point-normal form to represent it, that is, a three-dimensional coordinate , and the spatial normal vector normal=(a,b,c), obtaining the plane equation , where a, b, and c are the three components of the normal vector of the plane, respectively representing the components of the plane normal vector on the x-axis, y-axis, and z-axis.

[0110] Extract the vertical vector through vanishing point detection for all training photos, and convert the vertical vector to the Normal in the world coordinate system with the help of the pose matrix of the camera:

[0111] ,

[0112] Use the RANSAC (Random Sample Consensus algorithm) method to find the optimal vertical vector from multiple estimated vertical vectors to fit the normal=(a,b,c) of the entire three-dimensional scene.

[0113] Perform a dot product operation on each camera position and the obtained normal to obtain the projection length S of each camera position on this vector t , in order to ensure that each camera position is on this horizontal plane, subtract a distance margin margin from the minimum S t to obtain the minimum projection length S in this direction 分钟 , and obtain the plane point through to obtain a plane . .

[0114] 1. Obtain the water surface intersection point;

[0115] When simulating floods, for rays ray_s, ray_d and plane center, normal, through the formula , the depth value dep from the intersection point of the ray and the plane to the camera position is calculated. When dep is positive, it indicates that the ray intersects the plane; when dep is less than 0, the ray has no valid intersection with the plane (not considered); when dep is 0, the ray starting point is on the plane, and the visualization effect diagram of dep as shown in Figure 4 is obtained; then it is compared with the depth of each pixel predicted by the network model, and the schematic diagram as shown in Figure 5 is obtained.

[0116] First, when dealing with the situation where there is no valid intersection between the ray and the horizontal plane, that is, when dep is less than zero, the depth of the rays that have no valid intersection with the horizontal plane is set to the maximum value in depth. At this time, it can be directly judged whether there is a closer intersection caused by other objects or surfaces before each ray intersects with the horizontal plane by dep being less than depth, and a water surface mask is_water is obtained and visualized to get a black and white picture as shown in Figure 6 , where 0 is black and 1 is white (the value of 1 indicates that the ray first intersects with the water surface; the value of 0 indicates that the ray has intersected with other objects before reaching the water surface).

[0117] By this water body mask is_water, only the data where the ray first intersects with the water surface is retained, which helps the subsequent processing steps (such as rendering the water surface effect, calculating reflection and refraction) to be more efficient and accurate.

[0118] 2. Simulate the water surface waves at the intersection point;

[0119] Through the formula , the intersection point coordinates of the rays that first intersect with the water surface on the water surface are calculated, and the intersection point of the ray and the water surface obtained at this time is called .

[0120] The mapping of the water surface waves adopts a wave simulation algorithm in computer graphics. It passes through the water surface intersection point , uses the inverse fast Fourier transform to convert the frequency domain information to the spatial domain, and obtains the actual shape and normal vector of the waves, so that is obtained.

[0121] After obtaining the water surface intersection point , the wave normal , and the ray incident direction ray_d, through the formula , the reflection ray direction of the water surface intersection point is calculated.

[0122] Of course, when light hits the water surface, in addition to specular reflection, scattering will also occur, and is expanded into a matrix with the same number of rays:

[0123] ,

[0124] where the symbol represents the ellipsis in the matrix, that is, the repeated elements in the matrix , and the dimension of matrix N’ is , the parameter O represents the real number field, that is, the elements in the matrix are real numbers; u represents the number of rows of the matrix.

[0125] Construct the orthogonal basis vectors (Y) and (X), calculate the scattering direction to obtain the main direction and four scattering directions, and combine them into a tensor to obtain the scattering direction:

[0126] ,

[0127] where L represents the original incident ray direction; Lx0, Lx1 and Ly0, Ly1 respectively represent the four scattering directions under the action of water surface waves, which are the scattering components generated by the interaction between the wave surface and the incident ray.

[0128] Perform unit vectorization on each scattering direction:

[0129] ,

[0130] where Sk,m represents the m-th direction of the k-th ray in tensor S.

[0131] And initialize the weight matrix Calculate the weight of the side scattering direction Keep the weight of the main direction as 1, and set the weight of the side scattering direction to Perform weight normalization so that the sum of the weights of each ray is 1.

[0132] That is: ,

[0133] In this way, the obtained scattering direction S and the weights W in each direction are recorded. Take the scattering direction S and as the direction and starting point of the ray, and start from the step of obtaining the ray again to get the pixel value C_ref of the water surface reflection and scattering. Note that since each point has 5 scattering directions S, so this C_ref is no longer the pixel value of each point on the corresponding image, but the pixel value of a total of 5 directions including the scattering and incident directions. Therefore, at this time, according to the weight Wi,j obtained for each ray, these scattering rays representing the waves are superimposed to obtain the water surface reflection simulating the wave effect As Figure 7 shown

[0134] 3. Water surface reflection simulation;

[0135] Calculate the reflection coefficient M of the water surface according to the Fresnel effect:

[0136] , , ,

[0137] wherein, is the reflection coefficient of vertically polarized light, is the reflection coefficient of horizontally polarized light; is the refraction angle, is the incident angle.

[0138] The reflected ray directions and , obtain the incident angle θ by dot product summation, and obtain the reflectivity .

[0139] Then calculate the water surface color according to the formula as follows shown, wherein, Figure 8 is the preset water body color.

[0140] Flood simulation is obtained by fusing the water surface mask with the original scene color C as Figure 9 shown.

[0141] Among them, scene fusion includes fusing the final scenes obtained in the previous two steps, and simulating effects such as darkening the sky.

[0142] The present invention, through three-dimensional reconstruction technology, can intuitively display the impact of meteorological disasters on the environment, enabling users to visually feel the disaster situation, greatly enhancing the visualization effect and dissemination efficiency of data, so as to solve the problem that the traditional display of meteorological disasters mainly relies on texts and two-dimensional meteorological charts, with insufficient intuitiveness and interactivity.

[0143] With the new ray sampling method, the reconstructed 3D model has better quality and higher efficiency: Using the Code operator to inline the cuda kernel function of ray sampling into Python. Different from the way of pytorch to extend its own operators to cuda, in jittor, there is no need to create a new data type for the meta-operator, but use the data type of C++ itself. Jittor can automatically convert the C++ data type into the meta-operator, which speeds up the running time. At the same time, the xyzs and dirs are encoded, and a custom Code operator is created in the same way during the volume rendering process, and the obtained effect is also obvious.

[0144] As shown in Table 1 below, the present invention only needs a few minutes of inference time to achieve the effect of several hours on pytorch.

[0145] Table 1. Comparison table of inference efficiency between ClimateNeRF and the present invention

[0146]

[0147] The above are only the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and improvements, and can be within the scope of the concept described herein, through the above teachings or the technology or knowledge in the relevant field. Any changes and modifications made by those skilled in the art without departing from the spirit and scope of the present invention shall fall within the protection scope of the appended claims of the present invention.

Claims

1. A deep learning three-dimensional climate simulation method for adding climate to a real scene, characterized by: The simulation method comprises: Scene reconstruction steps: input the real scene image into the jittor framework, emit a ray for each pixel of the real scene image, sample each acquired ray, obtain the sampling point xyzs and direction dirs, encode the sampling point xyzs and direction dirs using the instantNGP encoder based on jittor implementation, and send them to the neural radiation field to predict the color value of each sampling point and density value , substitute the parameters predicted by the neural radiation field into the rendering equation and then input it into the volume rendering, so as to preserve the color, depth and opacity of the sampling point and realize scene reconstruction; Meteorological simulation steps: input meteorological disaster pictures into the ResUnet framework to segment meteorological disaster elements, so as to reconstruct the meteorological scene corresponding to the meteorological disaster; A scene fusion step, fusing the final scene obtained in the scene reconstruction step and the meteorological simulation step to obtain a climate simulation map; The ray sampling includes: Divide the space into a three-dimensional grid consisting of a first grid and a second grid, when the first sampling step is used, use the first grid to capture the main structure of the scene, when the second sampling step is used, use the second grid for sampling to capture the details of the scene, and associate the first grid and the second grid with the grid density, and concentrate the sampling points in the grids with a density value greater than a set value to ensure that more sampling is performed in the area with a density value greater than the set value, and the first sampling step is greater than the second sampling step; Calculate the position relationship between the image scene frame bounding box and the ray, and get ,when Greater than , the ray and the bounding box have no intersection; if If it is less than 0, the ray is behind the bounding box and there is no intersection. The intersection between the ray and the bounding box is recorded. and , set the ray starting point to , when t is greater than Or the sampling ends when the sampling point reaches the preset value, t represents the step length parameter of the ray along its path, Indicates the relationship between the minimum value of the bounding box in the x-axis direction and the position of the ray. Indicates the relationship between the maximum value of the bounding box in the x-axis direction and the position of the ray. and Respectively represent the minimum and maximum values ​​of the bounding box in the x-axis direction, Indicates the coordinate of the starting point of the ray on the x-axis, Represents the component of the ray's direction vector on the x-axis; Calculate the sampling step size to select the first grid or the second grid, control the sampling point to always be in the area where the density value is greater than the set value according to the density distribution in the grid, record the t value of the nth sampling point position on each ray into the ts array, and then obtain the three-dimensional coordinates of the sampling point xyzs and the corresponding direction dirs, where n represents the number of sampling points and ts represents the array; The meteorological simulation step includes: sandstorm meteorological simulation and flood meteorological simulation; The sandstorm meteorological simulation includes: constructing the sandstorm color and sandstorm density according to the Lambert-Beer law based on the depth, opacity and cumulative effect function, integrating them according to the volume rendering, obtaining a sandstorm scene color value picture of only the sandstorm, and combining the picture with the sandstorm climate simulation; The flood meteorological simulation includes: calculating the water surface mask of the image and the depth value at the intersection according to the center point and normal vector of the depth and horizontal plane, obtaining the water surface intersection, simulating the wave points and wave normals required for the water surface waves at the water surface intersection by the spectral method, calculating the reflection coefficients of these wave points on the water surface by the Fresnel effect, and sending the obtained reflected rays to ray sampling to obtain the reflection color of the intersection of the horizontal plane of the image, and obtaining the flood climate simulation according to the water surface mask, ray reflectivity, water body color and image color composition.

2. A deep learning three-dimensional climate simulation method for adding climate to a real scene according to claim 1, characterized in that: Each pixel of the real scene image emits a ray including: Use deep learning to learn the three-dimensional scene where the real scene picture is located. According to the batch size, randomly select each picture. Get the color value of the pixel , and the pixel position xy in the pixel coordinate system, Indicates the picture, Indicates the color value of the pixel; Get ray-related information from the corresponding camera pose matrix T, which includes the rotation matrix R and the translation vector trans, which is the position of the camera; Connect the camera's position coordinates to a pixel on the plane to determine a ray, use the translation vector trans as the ray's starting point ray_o, and calculate the ray's direction through the camera's rotation matrix R, principal point position p, resolution res, focal length f, pixel position xy, and translation vector trans. , and finally we get the ray equation rays(trans)=ray_o+trans·ray_d, where normalize means vector normalization.

3. The deep learning three-dimensional climate simulation method for adding climate to a real scene according to claim 1, characterized in that: The encoding of the sampling point xyzs and the direction dirs using the instantNGP encoder based on jittor specifically includes: For the sampling point xyzs, multi-resolution hash coding is used to encode the three-dimensional space at sixteen resolution levels from low to high. A learnable two-dimensional feature is set for each point at each resolution and stored in a hash table at each resolution. Then, for a given input coordinate xyz, the nearest eight vertices are found at each resolution, their feature values ​​are trilinearly interpolated, and the feature values ​​are combined to obtain a 32-dimensional feature vector. The third-order spherical harmonic function encoding is used for the directional dirs, and the directional dirs are fitted into a 16-dimensional feature vector.

4. The deep learning three-dimensional climate simulation method for adding climate to a real scene according to claim 3, characterized in that: The color value of each sampling point is predicted by the neural radiance field and density value Specifically include: The 32-dimensional features obtained after encoding the sampling point xyzs are input into the first MLP network to obtain the density value of each sampling point At the same time, the view direction is encoded and connected to the output of the first MLP network and then input into the second MLP network to obtain the predicted value of the color. The activation function Sigmoid is used for activation to obtain the color value of each sampling point. ; The color value of each sampling point on each ray and density value The final calculated pixel color is integrated through volume rendering. Get the color value of the pixel Do a loss calculation to get the difference between the real image and the image generated by the network.

5. The deep learning three-dimensional climate simulation method for adding climate to a real scene according to claim 1, characterized in that: The sandstorm weather simulation specifically includes: The density of dust storms in free space is simulated by adding a non-negative constant to the density of free space, and the predicted color, depth, and opacity of each pixel are calculated. and ; Will get As each ray To point The distance As a cumulative effect of the sandstorm, the sandstorm color and dust storm density As the neural radiance field output, let the volume rendering follow Integrate to get the color value C of the scene with only the sandstorm dust ; The obtained scene color value C dust Through a shallow sampling of the neural radiation field, C dust As output, a volume rendering is performed again on the original color to obtain a sandstorm climate simulation scene.

6. The deep learning three-dimensional climate simulation method for adding climate to a real scene according to claim 2, characterized in that: The method of calculating the water surface mask of the image and the depth value at the intersection point according to the depth and the center point of the horizontal plane and the normal vector, and obtaining the water surface intersection point specifically includes: In three-dimensional space, a point method is used to represent a plane, that is, a three-dimensional coordinate , space normal vector , and we get the plane equation , where a, b, c are the three components of the plane's normal vector, representing the components of the plane's normal vector on the x-axis, y-axis, and z-axis respectively; By formula Calculate the depth value dep from the intersection of the ray and the plane to the camera pose, and compare it with the predicted depth depth of each pixel. When dep is greater than 0, it indicates that the ray intersects the plane. When dep is less than 0, it indicates that there is no valid intersection between the ray and the plane. When dep is 0, the starting point of the ray is on the plane. Represents rays; When dep is less than 0, the ray depth is set to the maximum value in depth. By judging whether each ray has a closer intersection point caused by other objects or surfaces before intersecting the horizontal plane, a water surface mask is_water is obtained, and a black and white picture is obtained by visualization. The water surface mask is_water is used to retain the data where the ray that first intersects the water surface satisfies the judgment result that dep is less than depth.

7. A deep learning three-dimensional climate simulation method for adding climate to a real scene according to claim 6, characterized in that: The method of simulating the wave points and wave normals required for the water surface waves from the water surface intersection points by the spectral method specifically includes: By formula Calculate the coordinates of the intersection point on the water surface of the ray that first intersects the water surface. is the intersection point of the ray and the water surface; pass Use inverse fast Fourier transform to convert frequency domain information to spatial domain to obtain the actual shape and normal vector of the wave and get the wave normal ; By formula Calculate the direction of the reflected ray at the intersection of the wave and water surface, and Expanded to a matrix with the same number of rays, expressed as ,in, Represents an ellipsis in a matrix, i.e., repeated elements in the matrix ,matrix The dimension is , parameter O represents the real number domain, that is, the elements in the matrix are real numbers, and u represents the number of rows in the matrix; Construct orthogonal basis vectors, calculate the scattering direction to get the main direction and four scattering directions, and combine them into a tensor to get the scattering direction as , and each scattering direction is unit vectorized to , where L represents the original incident ray direction, , and , They represent the four scattering directions under the action of water surface waves, which are the scattering components generated by the interaction between the wave surface and the incident rays; Calculate the weight W and normalize it so that the sum of the weights of each ray is 1. Record the obtained scattering direction S and the weight W of each direction. As the direction and starting point of the ray, the pixel value C_ref of the water surface reflection and scattering is obtained; According to the weight obtained for each ray, these scattered rays representing waves are superimposed to obtain the water surface reflection simulating the wave effect. .

8. The deep learning three-dimensional climate simulation method for adding climate to a real scene according to claim 7, characterized in that: The calculation of the reflection coefficients of these wave points on the water surface by the Fresnel effect and the sending of the obtained reflected rays to the ray sampling to obtain the reflection color of the intersection of the horizontal plane of the picture specifically includes: Calculate the reflection coefficient of the water surface based on the Fresnel effect , where M represents the reflection coefficient, is the reflection coefficient of vertically polarized light, is the reflection coefficient of horizontally polarized light, is the angle of refraction, is the angle of incidence; The reflected ray direction and The angle of incidence is obtained by summing the dot product , and find the reflectivity ; According to the formula Calculate the water surface color , It is the preset water color; The flood climate simulation is obtained by merging the water surface mask with the original scene color.

Images