Unmanned aerial vehicle aerial image synthesis method and system, computer device and storage medium

Abstract

The application belongs to the technical field of computer graphics and computer vision, and discloses a UAV aerial image synthesis method and system, computer equipment and a storage medium, comprising: acquiring a plurality of two-dimensional images of UAV aerial photography; performing three-dimensional reconstruction of a scene according to the plurality of two-dimensional images to obtain a sparse three-dimensional point cloud model of the scene, and sampling in the sparse three-dimensional point cloud model to obtain a plurality of sampling points; acquiring training data of each sampling point, and training a preset complete scene neural radiance field network model according to the training data of each sampling point to obtain a trained complete scene neural radiance field network model; inputting a set view point direction into the trained complete scene neural radiance field network model to obtain a scene image under the set view point direction through rendering. The application has a faster rendering speed, can synthesize a scene in a large-scale range, and can also realize high-quality view rendering under an arbitrary view point.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer graphics and computer vision, and relates to a UAV aerial image synthesis method and system, a computer device and a storage medium. BACKGROUND

[0002] With the rapid development of science and technology, the business demand for 3D reconstruction technology in the fields of city development, cultural relic protection, virtual reality and industrial geographic survey is becoming stronger and stronger. However, the traditional 3D reconstruction method needs manual measurement of related data, and then professional software is used to draw 3D graphics, which is time-consuming and laborious. In recent years, the UAV technology has gradually become the main tool for engineers to obtain aerial images or video images due to its low cost, high efficiency and portability. At present, the research on three-dimensional reconstruction based on the sequence images captured by the UAV is becoming mature. Among them, the view synthesis based on images is an important problem in the field of computer graphics and computer vision. Specifically, a number of images taken from known shooting viewpoints are used as input, and the geometric, appearance, lighting and other properties of the three-dimensional object or scene shot by these images are expressed, so that the images of other unshot viewpoints can be synthesized, and finally a drawing result with high realism is obtained. Compared with the traditional three-dimensional reconstruction combined with graphic drawing process, this kind of method can obtain a photo-level realistic synthesis result.

[0003] Pix4D, VisualSFM, Smart3D and ColMap software based on the Structure From Motion (SFM) technology use the SFM method to solve the camera pose and the sparse point cloud in the three-dimensional space from the input aerial image sequence, then perform dense processing on the sparse point cloud, and then generate triangular facets by grid reconstruction on the dense point cloud, and finally perform texture mapping on the grid to obtain a three-dimensional map with texture information. Although the overall process of the method based on the SFM technology is mature, the huge amount of algorithm calculation makes it have high requirements for hardware configuration, especially for high-resolution aerial image sequences. In order to maintain the reconstruction result of the original resolution, the processing time will be very long, and the number of synthesized viewpoints is also very limited.

[0004] Synthesizing new scene views from a sparse set of captured images is a long-standing problem in computer vision and a prerequisite for many AR and VR applications. Although classic techniques have addressed this problem using either structure-from-motion or image-based rendering, there are still difficulties in explicit modeling, low accuracy of three-dimensional reconstruction and poor quality of rendered images. SUMMARY

[0005] The purpose of the present application is to overcome the above-mentioned shortcomings of the prior art, and to provide a UAV aerial image synthesis method and system, a computer device and a storage medium.

[0006] To achieve the above object, the application adopts the following technical solutions to achieve it:

[0007] In one aspect of the application, a UAV aerial image synthesis method comprises:

[0008] Obtaining a plurality of two-dimensional images taken by a UAV;

[0009] Reconstructing a three-dimensional scene from the plurality of two-dimensional images to obtain a sparse three-dimensional point cloud model of the scene, and sampling the sparse three-dimensional point cloud model to obtain a plurality of sampling points;

[0010] Obtaining training data for each sampling point, and training a preset complete scene neural radiance field network model according to the training data for each sampling point to obtain a trained complete scene neural radiance field network model; the training data includes sampling point coordinates, sampling point viewpoint direction, sampling point implicit scene lighting visibility feature vector, and sampling point actual color value;

[0011] Inputting a set viewpoint direction into the trained complete scene neural radiance field network model to obtain a scene image under the set viewpoint direction through rendering.

[0012] Optionally, the obtaining of the plurality of two-dimensional images taken by the UAV comprises:

[0013] Obtaining a plurality of two-dimensional images taken by the UAV at a fixed height, different positions and different angles, and the overlapping degree of adjacent two-dimensional images is greater than 80%.

[0014] Optionally, the three-dimensional reconstruction of the scene from the plurality of two-dimensional images comprises:

[0015] A colmap image reconstruction method of motion recovery structure is used to reconstruct the three-dimensional scene.

[0016] Optionally, the complete scene neural radiance field network model comprises a foreground neural radiance field network and a background neural radiance field network;

[0017] The foreground neural radiance field network comprises a foreground volume density synthesis network and a foreground color synthesis network; the foreground volume density synthesis network is: The foreground color synthesis network is:

[0018] Wherein, σ(t) represents a volume density function; z(t) represents a feature vector related to position coding; represents a volume density synthesis network; γ x represents position coding; represents the foreground image color estimated by the foreground color composition network, r(t) = o + td represents emitting a ray along the light ray origin, t e (0, t') represents the scene boundary as an interior unit sphere, T(t) represents the cumulative transparency along the viewpoint camera ray, and the calculation formula is represents, t represents the distance along the light ray r from the light ray origin, c i (t) represents the radiance, represents the foreground color composition network, γ d (d) represents the view direction encoding, represents the implicit lighting visibility feature vector encoding of the i-th image, a represents the lighting visibility feature vector;

[0019] The background neural radiance field network comprises a background volume density composition network and a background color composition network; the background volume density composition network is: The background color composition network is:

[0020] Wherein, represents the background volume density composition network, represents the background image color estimated by the background color composition network, t e (t', ∞) represents the outside of the unit circular sphere,

[0021] The rendering function of the complete scene neural radiance field network model is:

[0022]

[0023] Wherein, C i (r) is the synthesized color value of the complete scene neural radiance field network model, (i) is the foreground neural radiance field synthesis color value, (ii) is the synthesis coefficient, and (iii) is the background neural radiance field synthesis color value.

[0024] Optionally, obtaining the sampling point implicit scene lighting visibility feature vector of each sampling point comprises:

[0025] The opacity α and the cumulative transparency T of the camera ray of the sampling point along the viewpoint direction are obtained by using the following formula:

[0026]

[0027] α i = 1 - exp (- σ i δ i )

[0028] Wherein, δ i = t i+1 -t i represents the distance between adjacent sampling points, σ ia volume density representing a viewpoint direction; a sampling point implicit scene light visibility feature vector of a sampling point is obtained using the following formula

[0029]

[0030] wherein a is the light visibility feature vector.

[0031] Optionally, the sampling in the sparse three-dimensional point cloud model comprises:

[0032] According to the sparse three-dimensional point cloud model, the near boundary and the far boundary of the scene are obtained, and uniform sampling is performed between the near boundary and the far boundary of the scene to obtain the coarsely sampled sampling points;

[0033] The coarsely sampled sampling points are input into a preset complete scene neural radiance field network model to obtain a probability density distribution function of color;

[0034] According to the probability density distribution function of color, fine sampling is performed in a region where the probability value is greater than a preset threshold to obtain fine sampled sampling points;

[0035] The coarsely sampled sampling points and the fine sampled sampling points are combined to obtain the final sampling points.

[0036] Optionally, when the preset complete scene neural radiance field network model is trained according to the training data of each sampling point, a residual loss function L of the following formula is used for training:

[0037]

[0038] wherein R represents the total number of up-sampled points on the light of each batch of training, the color estimation value obtained by inputting the coarsely sampled sampling points into the complete scene neural radiance field network model, the color estimation value obtained by inputting the fine sampled sampling points into the complete scene neural radiance field network model, and C(r) represents the actual color value of the sampling point.

[0039] In a second aspect of the present application, an unmanned aerial vehicle aerial image synthesis system comprises:

[0040] An acquisition module is configured to acquire a plurality of two-dimensional images taken by an unmanned aerial vehicle;

[0041] A sampling module is configured to perform three-dimensional reconstruction of a scene according to the plurality of two-dimensional images to obtain a sparse three-dimensional point cloud model of the scene, and perform sampling in the sparse three-dimensional point cloud model to obtain a plurality of sampling points.

[0042] The training module is configured to obtain training data of each sampling point, and train a preset complete scene neural radiance field network model according to the training data of each sampling point, to obtain a trained complete scene neural radiance field network model; the training data comprises a sampling point coordinate, a sampling point viewpoint direction, a sampling point implicit scene light visibility feature vector and a sampling point actual color value;

[0043] The rendering module is configured to input a set viewpoint direction into the trained complete scene neural radiance field network model, and obtain a scene image under the set viewpoint direction through rendering.

[0044] In a third aspect, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the UAV aerial image synthesis method when executing the computer program.

[0045] In a fourth aspect, a computer readable storage medium stores a computer program, and the computer program implements the steps of the UAV aerial image synthesis method when executed by a processor.

[0046] Compared with the prior art, the present application has the following beneficial effects:

[0047] The UAV aerial image synthesis method of the present application obtains a plurality of two-dimensional images taken by a UAV, reconstructs a three-dimensional scene from the plurality of two-dimensional images to obtain a sparse three-dimensional point cloud model of the scene, samples the sparse three-dimensional point cloud model to obtain a plurality of sampling points, obtains training data of each sampling point, trains a preset complete scene neural radiance field network model according to the training data of each sampling point, and finally obtains a scene image under a set viewpoint direction through the trained complete scene neural radiance field network model. Based on the trained complete scene neural radiance field network model, view rendering under any viewpoint can be realized. Compared with the prior art, the network training time is shorter than the scene three-dimensional model reconstruction time, the rendering speed is faster, large-scale scenes can be synthesized, high-quality view rendering under any viewpoint can be realized, and real high-quality new views can be rendered using the pre-trained complete scene neural radiance field network model. Moreover, the neural network can be trained offline, real-time rendering of new viewpoints can be realized, and the synthesis range of new viewpoints is not limited, so the application prospect is better. At the same time, large-scale scene reconstruction using neural rendering can be accelerated using a GPU device, which is very suitable for large-scale scene reconstruction with large amounts of data. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1A flow chart of a UAV aerial image synthesis method according to an embodiment of the present application;

[0049] Figure 2 A detailed flow chart of a UAV aerial image synthesis method according to an embodiment of the present application;

[0050] Figure 3 A schematic diagram of an implicit scene lighting visibility feature vector network framework according to an embodiment of the present application;

[0051] Figure 4 A schematic diagram of a complete scene neural radiance field network model overall framework according to an embodiment of the present application;

[0052] Figure 5 A schematic diagram of a reverse sphere parameterization manner principle for segmenting foreground and background according to an embodiment of the present application;

[0053] Figure 6 A schematic diagram of a scene reconstruction result under a certain view angle generated by the method according to an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to enable persons skilled in the art to better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should be within the scope of protection of the present application.

[0055] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0056] The present application will be described in further detail below with reference to the drawings:

[0057] In the field of large-scale high-quality UAV aerial map synthesis, traditional three-dimensional reconstruction methods require three-dimensional modeling of the entire large-scale scene, which is time-consuming, and the accuracy of the synthesized three-dimensional model depends on the number of input views. The more images collected, the more accurate the model established. However, the increase in data volume results in a long time for three-dimensional reconstruction of the scene, and the reconstructed model cannot synthesize images within any view range, but only virtual view images with fixed angles, thus having great limitations.

[0058] Moreover, synthesizing new scene views from a sparse set of captured images is a long-standing problem in computer vision and a prerequisite for many AR and VR applications. Although classic techniques have addressed this problem using motion-based or image-based rendering, significant progress has recently been made in this field due to the introduction of neural radiance fields (NeRF), which use neural networks to model the radiance field and density of a scene. Then, new views are synthesized using volume-based rendering techniques, which have shown unprecedented realism on a range of challenging scenes.

[0059] In recent years, neural rendering techniques have become a hot research topic due to their ability to implicitly reconstruct scenes using neural networks, resulting in higher quality and faster image rendering based on volume-based rendering principles. With the rapid development of deep learning technology, many deep learning-based methods have been proposed to further improve the accuracy and realism of view synthesis through data-driven approaches. Large-scale high-quality real scene map synthesis based on neural radiance fields involves training a neural radiance field network to reconstruct the scene. Once the network is trained, it can be used to render views from any viewpoint. Compared to traditional methods that require pre-established three-dimensional models, the network training time is shorter, and the pre-trained network can be used to render high-quality new views with realistic quality. Since the neural network can be trained offline, it can achieve real-time rendering of new viewpoints, and the synthesis range of new viewpoints is not limited, making it a promising application.

[0060] To this end, the application discloses a UAV aerial image synthesis method, which is used for reconstructing a dense scene view according to a sparse two-dimensional image sequence obtained by UAV aerial photography. Since a large-scale scene needs to be reconstructed, if all scene contents are reconstructed by a neural radiance field, a scene reconstruction with a large depth range may have a detail blur problem, so the foreground and the background in the image need to be modeled respectively; and considering that the light will change when the UAV is aerially photographed, the obtained static images are encoded by using an implicit light visibility vector, the foreground and the background neural radiance field models with variable light are synthesized, then the two obtained models are spliced to obtain a neural radiance field model of the reconstructed scene, and a neural rendering technology is used to realize large-scale high-quality UAV aerial map image synthesis.

[0061] The neural rendering method based on the neural radiance field converts a scene needing three-dimensional modeling display into an implicit function simulating real imaging, estimates the color and density features of a picture along a line of sight for rendering, and thus obtains a reconstructed result very consistent with the original picture. By means of the method, a trained model can be used to realize large-scale high-quality image synthesis with variable light at an arbitrary viewpoint.

[0062] Referring to Figure 1 and 2 In an embodiment of the application, a UAV aerial image synthesis method is provided, which is suitable for large-scale high-quality UAV aerial image synthesis, and a computer simulation is used to generate a UAV aerial rendering image. In the embodiment, the UAV aerial image synthesis method specifically comprises the following steps:

[0063] S1: acquiring a plurality of two-dimensional images obtained by UAV aerial photography.

[0064] The acquiring of the plurality of two-dimensional images obtained by UAV aerial photography comprises: acquiring a plurality of two-dimensional images obtained by UAV aerial photography at a fixed height, different positions and different viewpoints, and the overlapping degree of adjacent two-dimensional images is greater than 80%. The plurality of two-dimensional images obtained by UAV aerial photography is generally a two-dimensional image sequence N is the number of two-dimensional images. Specifically, when a large-scale scene picture is aerially photographed by a UAV, a flight path of the UAV is planned in advance, so that the UAV flies at a fixed height, the overlapping degree of adjacent images is greater than 80%, the camera tilt angle is 45°, and N two-dimensional images obtained by the UAV at different positions and different viewpoints are acquired to form a two-dimensional image sequence

[0065] S2: three-dimensional reconstruction of a scene is performed according to the plurality of two-dimensional images, a sparse three-dimensional point cloud model of the scene is obtained, and a plurality of sampling points are obtained by sampling in the sparse three-dimensional point cloud model.

[0066] The three-dimensional reconstruction of the scene from the plurality of two-dimensional images comprises a colmap image reconstruction method of motion recovery structure.

[0067] Specifically, the two-dimensional image sequence obtained by the unmanned aerial vehicle aerial photography is subjected to image data preprocessing, and a colmap image reconstruction program of motion recovery structure technology is used to perform feature extraction, feature matching and sparse reconstruction on the image sequence to obtain the internal and external parameters of the camera, sparse 3D position points and near and far plane parameters, wherein the near and far plane parameters include the near boundary and the far boundary of the scene.

[0068] Then, sampling is performed in the sparse three-dimensional point cloud model, which generally comprises coarse sampling and fine sampling. Specifically, the near boundary and the far boundary of the scene are obtained according to the sparse three-dimensional point cloud model, and uniform sampling is performed between the near boundary and the far boundary of the scene to obtain the sampling points of coarse sampling; the sampling points of coarse sampling are input into a preset complete scene neural radiance field network model to obtain a probability density distribution function of color; fine sampling is performed in a region where the probability value is greater than a preset threshold according to the probability density distribution function of color, that is, more sampling points are placed at positions with large probability to obtain the sampling points of fine sampling; and the sampling points of coarse sampling and the sampling points of fine sampling are combined to obtain the final sampling points.

[0069] S3: Obtain training data of each sampling point, and train the preset complete scene neural radiance field network model according to the training data of each sampling point to obtain a trained complete scene neural radiance field network model; the training data comprises sampling point coordinates, sampling point viewpoint directions, sampling point implicit scene light visibility feature vectors and sampling point actual color values.

[0070] Specifically, the sampling point coordinates and the sampling point viewpoint directions can be obtained through the sparse three-dimensional point cloud model, and the sampling point implicit scene light visibility feature vector is obtained through the following steps:

[0071] The opacity a and the cumulative transparency T of the camera ray along the viewpoint direction of the sampling point are obtained by using the following formula:

[0072]

[0073] a = 1 - exp(-a i a i a i )

[0074] Wherein, subscript i is the sampling point number.

[0075] The sampling point implicit scene light visibility feature vector of the sampling point is obtained by using the following formula:

[0076]

[0077] Wherein, a is the light visibility feature vector.

[0078] Specifically, the image dimension of the two-dimensional image is HxWx3, the dimension size of the sampling point implicit scene light visibility feature vector is set to HxW, for each two-dimensional image, the line connecting the optical center and the pixel point is a ray, n rays are selected at random, for each ray, the opacity a and the cumulative transparency T of the camera ray along the viewpoint p direction are calculated by using the following formula:

[0079]

[0080] a = 1 - exp(-a i a i a i )

[0081] Wherein, a i = t i+1 -t i Indicates the distance between adjacent sampling points, a i Indicates the volume density of the viewpoint p direction.

[0082] Then, the visibility feature vector of the camera ray along the viewpoint p direction is calculated by using the following formula:

[0083]

[0084] The two-dimensional image sequence Is input into the implicit scene light visibility feature vector network, and the network training is carried out by using the following formula:

[0085]

[0086] Wherein, L' represents the prediction error;​ represents the illumination visibility feature vector estimation value calculated through the network; and l(r) represents the actual illumination visibility.

[0087] Referring to Figure 3 , the implicit scene illumination visibility feature vector network is a fully connected neural network, the input is the position coordinates and the viewpoint direction, the network training uses the Adam optimization method, the network weight parameters are updated simultaneously through the back propagation algorithm, and the implicit scene illumination visibility feature vector set is obtained. Through the implicit scene illumination visibility feature vector network after training, the final implicit scene illumination visibility feature vector encoding of each two-dimensional image is obtained During the training of the implicit scene illumination visibility feature vector network, the light direction needs to be sampled according to the pixel point and the camera position, and the sampling is uniform sampling. The implicit scene illumination visibility feature vector encoding of each two-dimensional image obtained subsequently is spliced with each two-dimensional image and input into the preset complete scene neural radiance field network model for training. For example, for each two-dimensional image in the two-dimensional image sequence , the light ray connecting the optical center and the pixel point is a light ray, n light rays are selected at random, 16 points are uniformly sampled for each light ray, the position information of the sampling points is encoded and the viewpoint direction is encoded, and the implicit scene illumination visibility feature vector network is input to obtain N implicit scene illumination visibility feature vectors corresponding to the two-dimensional image sequence . Then, the complete scene neural radiance field network model is input to obtain the corresponding color value.

[0088] The above steps are to realize the feature vector encoding of different illumination visibility information on each pixel point corresponding to the light ray on the image with different viewpoint directions between scenes, to obtain the global illumination visibility information of the scene, and to input the neural radiance field network to further synthesize the neural radiance field with variable illumination.

[0089] Referring to Figure 4 , the complete scene neural radiance field network model includes a foreground neural radiance field network and a background neural radiance field network; the foreground neural radiance field network includes a foreground volume density synthesis network and a foreground color synthesis network; the foreground volume density synthesis network is: The foreground color synthesis network is: Wherein, σ(t) represents the volume density function; z(t) represents the position coding related feature vector; represents the volume density synthesis network; γ x represents the position coding; represents the foreground image color estimated by the foreground color synthesis network, r = o + t ddenotes emitting a ray along the ray origin, t e (0, t') denotes the scene boundary for the interior unit sphere, T(t) denotes the accumulated transparency along the view camera ray, and the calculation formula is denotes, t denotes the distance of the ray r from the ray origin, c i (t) denotes the radiance, denotes the foreground color synthesis network, γ d (d) denotes the view direction encoding, denotes the implicit lighting visibility feature vector encoding of the ith image, and a denotes the lighting visibility feature vector.

[0090] The background neural radiance field network comprises a background volume density synthesis network and a background color synthesis network; the background volume density synthesis network is: The background color synthesis network is: Wherein, denotes the background volume density synthesis network, denotes the background image color estimated by the background color synthesis network, t e (t', ∞) denotes the outside of the unit circular sphere,

[0091] The rendering function of the complete scene neural radiance field network model is:

[0092]

[0093] Wherein, C i (r) is the synthesized color value of the complete scene neural radiance field network model, (i) is the foreground neural radiance field synthesis color value, (ii) is the synthesis coefficient, and (iii) is the background neural radiance field synthesis color value.

[0094] In this embodiment, for constructing the preset complete scene neural radiance field network model, the following specific implementation steps are provided: the scene with different depth values in the scene is modeled separately, and when processing the part with larger depth value, the new view with high quality is synthesized for better processing of the detail part. Therefore, the scene space is divided into two parts, an interior unit sphere containing the foreground in the scene and an external volume containing the background in the scene. Among them, the interior unit sphere is modeled by the foreground neural radiance field, without additional parameterization, and the external unit sphere is modeled by the background neural radiance field, which needs to be parameterized. See Figure 5 Wherein, the inverse spherical parameterization process is as follows: the scene is represented in a unit circle S, and the inverse spherical parameterization is performed for the foreground and background division of the scene. The position point in the foreground of the scene can be represented as (x, y, z), and the position point in the background is represented as Wherein, r' is the radius of the unit circle S.

[0095]

[0096] The position encoding and direction encoding of the sampling points on the light rays in the unit circle are directly input into the foreground radiance field, and the foreground neural radiance field modeling can be represented as a function: MLP1(x, d) = (c, σ), where MLP is a fully connected network, x is a three-dimensional space coordinate, d is a two-dimensional view direction, c is a three-channel color output of the x position, and σ is a volume density at the x position, where the volume density of 0 represents that the space is not occupied, and 1 represents the object surface, which is used to process the foreground part of the scene shot by the camera, and the network structure thereof includes two MLP networks, one of which is a volume density synthesis network, and the other is a color synthesis network. The view rendering of a new view point is performed using the foreground neural radiance field, and when the camera light ray r of the known view point is given, the number of camera light rays r is determined by the number of pixel points of the known view. According to the light ray projection algorithm, the color estimation value of the light ray projection onto the new view is calculated by the following formula:

[0097]

[0098] The position information of the sampling points on the light rays after the inverse sphere parameterization preprocessing, that is, the sampling point coordinates and the sampling point view direction, is input into the background radiance field network, and the network structure thereof is a fully connected neural network, which is used to process the background part of the scene shot by the camera. Since the depth range of the background is much larger than that of the foreground, if the foreground radiance module is input, it may cause rough details to be synthesized, therefore, for the background part with a large depth value, it is input into the background radiance field for processing, and the network structure thereof includes two MLP networks, one of which is a volume density synthesis network, and the other is a color synthesis network. Specifically, the background neural radiance field modeling function is similar to the foreground modeling function, MLP2(x', d) = (c, σ). The difference lies in that the position points (x, y, z) on the light rays outside the unit circle need to be re-parameterized into a four-tuple (x', y', z', 1 / r), x' 2 +y′ 2 +z′ 2 = 1, where (x', y', z') is a unit vector in the same direction as (x, y, z), and 1 / r (0 < 1 / r < 1) is the inverse radius in the direction indicating the point r · (x', y', z') outside the sphere. The re-parameterized four-tuple is bounded, where (x', y', z') ∈ [-1, 1], and 1 / r ∈ [0, 1]. The four-tuple (x', y', z', 1 / r) obtained by re-parameterization is used to represent the points of the background part of the scene, and then the color and volume density of the corresponding pixel position points of the new view are calculated by inputting the background neural radiance field, and the new view rendering function is calculated by the following formula:

[0099]

[0100] The obtained implicit scene lighting visibility feature vector is taken as an input of a network branch to obtain a lighting-variable foreground and background neural radiance field, and a static scene three-dimensional geometric structure shared by all images is obtained. The color synthesis network of the lighting-variable foreground neural radiance field is as follows:

[0101]

[0102]

[0103] The color synthesis network of the lighting-variable background neural radiance field is as follows:

[0104]

[0105]

[0106] The neural radiance field fuses the foreground and the background to obtain a complete scene by using a volume rendering technique. Specifically, the foreground neural radiance field and the background neural radiance field are spliced to obtain a neural radiance field model of the entire scene, that is, the complete scene neural radiance field network model, and the rendering function of the complete scene neural radiance field network model is as follows:

[0107]

[0108] When the preset complete scene neural radiance field network model is trained according to the training data of each sampling point, a residual loss function L is used for training, and the residual loss function L is as follows:

[0109]

[0110] wherein R represents the total number of up-sampling points of a light ray in each batch of training, is a color estimation value of the complete scene neural radiance field network model obtained by inputting the coarse sampling point, is a color estimation value of the complete scene neural radiance field network model obtained by inputting the fine sampling point, and C(r) represents an actual color value of the sampling point.

[0111] Specifically, the complete scene neural radiance field network model adopts a backward propagation training method, including the following steps: (1) normalizing the training data; (2) inputting a training data into the complete scene neural radiance field network model, and calculating the output of the complete scene neural radiance field network model; (3) calculating the error between the actual output and the expected output of the complete scene neural radiance field network model, and adjusting the parameters of each layer in reverse layer by layer according to the Adam optimization method; (4) repeating steps (2) and (3) until all training data have been input; (5) calculating the cumulative total error between the actual output and the expected output of all training data, increasing the training number by 1, and ending the training if the total error is less than the set total error or the training number is greater than the set training number.

[0112] S4: inputting the set viewpoint direction into the trained complete scene neural radiance field network model to obtain a scene image under the set viewpoint direction by rendering.

[0113] Specifically, the set viewpoint direction is input into the trained complete scene neural radiance field network model to obtain a scene image under the set viewpoint direction. The dense view generation after the sparse view input neural network is realized, and finally a large-scale high-quality unmanned aerial vehicle scene image under any viewpoint is obtained.

[0114] In summary, the unmanned aerial vehicle aerial image synthesis method of the present application obtains a plurality of two-dimensional images by aerial photography of an unmanned aerial vehicle, reconstructs a three-dimensional scene according to the plurality of two-dimensional images to obtain a sparse three-dimensional point cloud model of the scene, samples the sparse three-dimensional point cloud model to obtain a plurality of sampling points, obtains training data of each sampling point, trains a preset complete scene neural radiance field network model according to the training data of each sampling point, and finally obtains a scene image under a set viewpoint direction through the trained complete scene neural radiance field network model. Based on the trained complete scene neural radiance field network model, view rendering under any viewpoint can be realized. Compared with the need to pre-establish a three-dimensional model of the scene, the network training time is shorter than the scene three-dimensional model reconstruction time, has a faster rendering speed, can synthesize a scene in a large-scale range, can realize high-quality view rendering under any viewpoint, and can realize real-level high-quality new views by using the pre-trained complete scene neural radiance field network model for rendering. Moreover, the neural network can be trained offline and can realize real-time rendering of new viewpoints, and the synthesis range of the new viewpoints is not limited, so it has a better application prospect. At the same time, the large-scale scene reconstruction by using neural rendering can be accelerated by using a GPU device, which is very suitable for large-scale scene reconstruction with large data volume.

[0115] Reference Figure 6In still another embodiment of the present application, the aerial sequence images of the tower shot by the unmanned aerial vehicle outdoors are input into the network proposed in the present application for training to obtain a complete tower scene reconstruction model, and the trained neural network can be used to realize rendering of the tower view at the acquired camera position.

[0116] It can be seen that under sparse input, the scene reconstruction result obtained by training the neural network proposed in the present application has certain improvement in the reconstruction result of the foreground and background parts with large depth values, relative to the original neural radiance field network, because the foreground and background of the large-scale scene are modeled separately.

[0117] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the apparatus embodiment, please refer to the method embodiments of the present application.

[0118] In still another embodiment of the present application, an unmanned aerial vehicle aerial image synthesis system is provided, which can be used to realize the unmanned aerial vehicle aerial image synthesis method described above. Specifically, the unmanned aerial vehicle aerial image synthesis system comprises an acquisition module, a sampling module, a training module and a rendering module.

[0119] The acquisition module is configured to acquire a plurality of two-dimensional images shot by the unmanned aerial vehicle; the sampling module is configured to perform three-dimensional reconstruction of the scene according to the plurality of two-dimensional images to obtain a sparse three-dimensional point cloud model of the scene, and sample in the sparse three-dimensional point cloud model to obtain a plurality of sampling points; the training module is configured to obtain training data of each sampling point, and train a preset complete scene neural radiance field network model according to the training data of each sampling point to obtain a trained complete scene neural radiance field network model; the training data comprises sampling point coordinates, sampling point viewpoint direction, sampling point implicit scene lighting visibility feature vector and sampling point actual color value; and the rendering module is configured to input a set viewpoint direction into the trained complete scene neural radiance field network model to obtain a scene image at the set viewpoint direction through rendering.

[0120] In a possible implementation, the acquiring a plurality of two-dimensional images shot by the unmanned aerial vehicle comprises: acquiring a plurality of two-dimensional images shot by the unmanned aerial vehicle at a fixed height, different positions and different viewpoints, and the overlap degree of adjacent shot two-dimensional images is greater than 80%.

[0121] In a possible implementation, the three-dimensional reconstruction of the scene according to the plurality of two-dimensional images comprises: performing three-dimensional reconstruction of the scene by a colmap image reconstruction method of motion recovery structure.

[0122] In a possible implementation, the complete scene neural radiance field network model comprises a foreground neural radiance field network and a background neural radiance field network; the foreground neural radiance field network comprises a foreground volume density synthesis network and a foreground color synthesis network; the foreground volume density synthesis network is: The foreground color synthesis network is: wherein σ(t) represents a volume density function; z(t) represents a position encoding related feature vector; denotes a volume density synthesis network; γ x denotes a position encoding; denotes a foreground image color estimated by the foreground color synthesis network, r(t) = o + td represents that a light ray is emitted along an origin of the light ray, t ∈ (0, t') represents a scene boundary that is an internal unit sphere, T(t) represents a cumulative transparency along a viewpoint camera light ray, and the calculation formula is denotes, t represents a distance of the light ray r from the origin of the light ray, c i (t) represents radiance, denotes the foreground color synthesis network, γ d (d) represents a viewing direction encoding, denotes an implicit lighting visibility feature vector encoding of the i-th image, a represents a lighting visibility feature vector; the background neural radiance field network comprises a background volume density synthesis network and a background color synthesis network; the background volume density synthesis network is: The background color synthesis network is: wherein, denotes a background volume density synthesis network, denotes a background image color estimated by the background color synthesis network, t ∈ (t', ∞) represents an external unit spherical ball, a rendering function of the complete scene neural radiance field network model is: wherein C i (r) is a synthesized color value of the complete scene neural radiance field network model, (i) is a foreground neural radiance field synthesized color value, (ii) is a synthesis coefficient, and (iii) is a background neural radiance field synthesized color value.

[0123] In a possible implementation, obtaining a sampling point implicit scene lighting visibility feature vector of each sampling point comprises: obtaining an opacity α and a cumulative transparency T of a camera light ray of the sampling point along a viewpoint direction by using the following formula: wherein δ i = t i+1 -t i denotes a distance between adjacent sampling points, σ i denotes a volume density along the viewpoint direction; and obtaining the sampling point implicit scene lighting visibility feature vector of the sampling point by using the following formula wherein a is the illumination visibility feature vector.

[0124] In a possible implementation, the sampling in the sparse three-dimensional point cloud model comprises: obtaining a near bound and a far bound of the scene according to the sparse three-dimensional point cloud model, and uniformly sampling between the near bound and the far bound of the scene to obtain coarsely sampled sampling points; inputting the coarsely sampled sampling points into the preset complete scene neural radiance field network model to obtain a probability density distribution function of color; according to the probability density distribution function of color, finely sampling in a region where a probability value is greater than a preset threshold to obtain finely sampled sampling points; and combining the coarsely sampled sampling points and the finely sampled sampling points to obtain final sampling points.

[0125] In a possible implementation, when the preset complete scene neural radiance field network model is trained according to the training data of each sampling point, a residual loss function L of the following formula is used for training: wherein R represents a total number of up-sampled points of each batch of rays during training, is a color estimation value obtained by inputting the coarsely sampled sampling points into the complete scene neural radiance field network model, is a color estimation value obtained by inputting the finely sampled sampling points into the complete scene neural radiance field network model, and C(r) represents an actual color value of the sampling point.

[0126] All relevant contents of each step of the foregoing embodiment of the unmanned aerial vehicle aerial image synthesis method can be cited to the function description of the function module corresponding to the unmanned aerial vehicle aerial image synthesis system in the embodiment of the application, and will not be repeated here.

[0127] The division of the modules in the embodiments of the application is illustrative, and is merely a logical function division. When actually implemented, another division manner can be used. In addition, each function module in each embodiment of the application can be integrated in one processor, or can be physically separated, or two or more modules can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software function module.

[0128] In still another embodiment of the present application, a computer device is provided, which comprises a processor and a memory, the memory being configured to store a computer program, the computer program comprising program instructions, and the processor being configured to execute the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc., which are the computing core and control core of the terminal, and are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in the computer storage medium to implement a corresponding method flow or a corresponding function; the processor in the embodiments of the present application can be used for the operation of the unmanned aerial vehicle aerial image synthesis method.

[0129] In still another embodiment of the present application, the present application further provides a storage medium, specifically a computer readable storage medium (Memory), which is a memory device in the computer device, and is used for storing programs and data. It can be understood that the computer readable storage medium herein can include the built-in storage medium in the computer device, and of course can also include the expansion storage medium supported by the computer device. The computer readable storage medium provides a storage space, which stores the operating system of the terminal. Moreover, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer readable storage medium herein can be a high-speed RAM memory, or a non-volatile memory such as at least one disk memory. One or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the unmanned aerial vehicle aerial image synthesis method in the above embodiments.

[0130] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0131] The present application is described in reference to the flowchart and / or block diagrams of the method, apparatus (system) and computer program product according to embodiments of the application. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing device or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0132] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0133] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0134] Finally, it should be noted that the above-mentioned embodiments are merely intended for describing the technical solutions of the present application, but not for limiting it. Although the present application is described in detail with reference to the above embodiments, those skilled in the field should understand that the specific embodiments of the present application can be modified or replaced equivalently without departing from the spirit and scope of the present application, and any modification or equivalent replacement without departing from the spirit and scope of the present application should be covered in the protection scope of the claims of the present application.

Claims

1. A method for synthesizing aerial images of a UAV, characterized in that, The method comprises the following steps: acquiring a plurality of two-dimensional images taken by a UAV; performing three-dimensional reconstruction of a scene according to the plurality of two-dimensional images to obtain a sparse three-dimensional point cloud model of the scene, and sampling in the sparse three-dimensional point cloud model to obtain a plurality of sampling points; acquiring training data of each sampling point, and training a preset complete scene neural radiance field network model according to the training data of each sampling point to obtain a trained complete scene neural radiance field network model; the training data comprises a sampling point coordinate, a sampling point viewpoint direction, a sampling point implicit scene light visibility feature vector, and a sampling point actual color value; inputting a set viewpoint direction into the trained complete scene neural radiance field network model to obtain a scene image under the set viewpoint direction through rendering; the complete scene neural radiance field network model comprises a foreground neural radiance field network and a background neural radiance field network; the foreground neural radiance field network comprises a foreground volume density synthesis network and a foreground color synthesis network; The foreground volume density synthesis network is: The foreground color synthesis network is: , ; wherein, denotes a volume density function; denotes a position encoding related feature vector; denotes a volume density synthesis network; denotes a position encoding; denotes a foreground image color estimated by the foreground color synthesis network, denotes emitting a ray along the ray origin, denotes a scene boundary as an interior unit sphere, denotes a cumulative transparency along the viewpoint camera ray, computed as denotes, denotes a distance along the ray r from the ray origin, denotes radiance, denotes a foreground color synthesis network, denotes a view direction encoding, denotes an implicit lighting visibility feature vector encoding for the i th image, a denotes a lighting visibility feature vector; The background neural radiation field network comprises a background volume density synthesis network and a background color synthesis network; the background volume density synthesis network is: , the background color synthesis network is: , ; wherein, represents a background volume density synthesis network, represents a background image color estimated by the background color synthesis network, represents the exterior of a unit sphere, ; the rendering function of the complete scene neural radiance field network model is: wherein, is a synthesized color value for the complete scene neural radiance field network model, i is a foreground neural radiance field synthesized color value, ii is a synthesized coefficient, iii is a background neural radiance field synthesized color value. 2.The UAV aerial image compositing method of claim 1, wherein, the acquiring of the plurality of two-dimensional images taken by the UAV comprises: acquiring a plurality of two-dimensional images taken by the UAV at a fixed height, different positions and different angles, and the overlapping degree of adjacent taken two-dimensional images is greater than 80%. 3.The UAV aerial image compositing method of claim 1, wherein, the three-dimensional reconstruction of the scene according to the plurality of two-dimensional images comprises: performing three-dimensional reconstruction of the scene by a colmap image reconstruction method of a motion recovery structure. 4.The UAV aerial image compositing method of claim 1, wherein, the acquiring of the sampling point implicit scene light visibility feature vector of each sampling point comprises: The opacity of the camera ray of the sampling point along the view direction is obtained using the following formula α and the accumulated transparency T : wherein, denotes the distance between adjacent sample points, denotes the volume density of the viewpoint direction; A sample point implicit scene lighting visibility feature vector for a sample point is obtained using the following equation : wherein, a is the lighted visibility feature vector. 5.The UAV aerial image compositing method of claim 1, wherein, the sampling in the sparse three-dimensional point cloud model comprises: obtaining a near boundary and a far boundary of the scene according to the sparse three-dimensional point cloud model, and uniformly sampling between the near boundary and the far boundary to obtain coarsely sampled sampling points; inputting the coarsely sampled sampling points into the preset complete scene neural radiance field network model to obtain a probability density distribution function of color; performing fine sampling in a region with a probability value greater than a preset threshold according to the probability density distribution function of color to obtain finely sampled sampling points; combining the coarsely sampled sampling points and the finely sampled sampling points to obtain final sampling points. 6.The UAV aerial image compositing method of claim 5, wherein, The residual loss function in the formula is used when training the preset complete scene neural radiance field network model according to the training data of each sampling point Training: wherein, R represents the total number of light ray up-sampling points of each batch of training, is the color estimation value obtained by inputting the sampling points of coarse sampling into the complete scene neural radiance field network model, is the color estimation value obtained by inputting the sampling points of fine sampling into the complete scene neural radiance field network model, represents the actual color value of the sampling points.

7. A UAV aerial image synthesis system, comprising: The method comprises the following steps: an acquisition module is configured to acquire a plurality of two-dimensional images taken by a UAV; a sampling module is configured to perform three-dimensional reconstruction of a scene according to the plurality of two-dimensional images to obtain a sparse three-dimensional point cloud model of the scene, and sample in the sparse three-dimensional point cloud model to obtain a plurality of sampling points; a training module is configured to acquire training data of each sampling point, and train a preset complete scene neural radiance field network model according to the training data of each sampling point to obtain a trained complete scene neural radiance field network model; the training data comprises a sampling point coordinate, a sampling point viewpoint direction, a sampling point implicit scene light visibility feature vector, and a sampling point actual color value; a rendering module is configured to input a set viewpoint direction into the trained complete scene neural radiance field network model to obtain a scene image under the set viewpoint direction through rendering; the complete scene neural radiance field network model comprises a foreground neural radiance field network and a background neural radiance field network; the foreground neural radiance field network comprises a foreground volume density synthesis network and a foreground color synthesis network; The foreground volume density synthesis network is: The foreground color synthesis network is: , ; wherein, denotes a volume density function; denotes a position encoding related feature vector; denotes a volume density synthesis network; denotes a position encoding; denotes a foreground image color estimated by the foreground color synthesis network, denotes emitting a ray along the ray origin, denotes a scene boundary as an interior unit sphere, denotes a cumulative transparency along the viewpoint camera ray, computed as denotes, denotes a distance along the ray r from the ray origin, denotes radiance, denotes a foreground color synthesis network, denotes a view direction encoding, denotes an implicit lighting visibility feature vector encoding for the i th image, a denotes a lighting visibility feature vector; The background neural radiation field network comprises a background volume density synthesis network and a background color synthesis network; the background volume density synthesis network is: , and the background color synthesis network is: , ; wherein, represents a background volume density synthesis network, represents a background image color estimated by the background color synthesis network, represents the exterior of a unit sphere, ; A rendering function of the complete scene neural radiance field network model is: wherein, is a synthesized color value for the complete scene neural radiance field network model, i is a foreground neural radiance field synthesized color value, ii is a synthesized coefficient, iii is a background neural radiance field synthesized color value.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor implements the steps of the unmanned aerial vehicle aerial image synthesis method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the steps of the unmanned aerial vehicle aerial image synthesis method according to any one of claims 1 to 6.

Images