Optimization Method for Oblique Photogrammetry LoD Model Based on Differentiable Rendering

Abstract

The present application discloses an optimization method for an oblique photography LoD model based on differentiable rendering, belonging to the technical field of data processing. The method includes: loading the model to be optimized in the oblique photography LoD model into the target scene, and loading the reference model into the reference scene. Based on the photography parameters of the oblique photography, rendering the target scene and the reference scene through a virtual camera to obtain a target image and a reference image, calculating the spatial error between the target image and the reference image, calculating the model optimization gradient corresponding to the spatial error through inverse rendering, and optimizing the model to be optimized based on the model optimization gradient to obtain an optimized model. The present application, in the LoD model after oblique photography modeling, takes a high-level high-precision model as a reference to optimize and iterate a low-level lower-precision model, reducing the spatial error between the low-level model and the high-level model, thereby improving the model accuracy of the low-level model.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular to an optimization method for an oblique photography LoD model based on differentiable rendering. Background Art

[0002] Oblique photography modeling uses sensors carried by drones to collect images and record aerial measurement information from vertical and four oblique angles respectively, and combines methods such as image segmentation and feature extraction for automatic modeling. In related technologies, oblique photography modeling uses the level of detail (LoD) technology to dynamically adjust the rendering fineness of the model based on different viewing angles and distances of the observer, so as to reduce the number of triangular faces in rendering and improve the rendering efficiency.

[0003] However, due to the large model noise during the oblique photography modeling process, at the same accuracy, the number of triangular faces, vertices, and material resolution required for the oblique photography model are much larger than those of the manual model expressing the same building. When the number of triangular faces of the oblique photography level of detail model is still larger than that of the manual model, the rendering result is more blurred compared to the manual model. This results in insufficient rendering accuracy of the oblique photography LoD model.

[0004] The above content is only used to assist in understanding the technical solution of this application, and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide an optimization method for an oblique photography LoD model based on differentiable rendering, aiming to solve the technical problem of insufficient rendering accuracy of the oblique photography LoD model.

[0006] To achieve the above purpose, this application provides an optimization method for an oblique photography LoD model based on differentiable rendering, and the method includes the following steps:

[0007] In the level of detail LoD model of oblique photography, obtain the model to be optimized and the reference model;

[0008] Load the model to be optimized into the target scene, and load the reference model into the reference scene;

[0009] Based on the photography parameters of the oblique photography, render the target scene and the reference scene through a virtual camera to obtain a target image and a reference image;

[0010] Calculate the spatial error between the target image and the reference image, and calculate the model optimization gradient corresponding to the spatial error through inverse rendering;

[0011] Based on the model optimization gradient, optimize the model to be optimized to obtain an optimized model.

[0012] In one embodiment, after the step of optimizing the model to be optimized based on the optimized gradient of the model, the method further includes:

[0013] Calculating an objective space error between the optimized model and a reference model;

[0014] When the objective space error is equal to or greater than a space error threshold, using the optimized model as the model to be optimized;

[0015] Jumping to execute the steps of loading the model to be optimized into a target scene and loading the reference model into a reference scene;

[0016] Alternatively, when the objective space error is less than the space error threshold, updating the optimized model to the LoD model of the oblique photography.

[0017] In one embodiment, the step of calculating a space error between the target image and the reference image and calculating an optimized gradient of the model corresponding to the space error through inverse rendering includes:

[0018] Identifying target pixel points of the target image and mapping them to reference pixel points in the reference image;

[0019] Calculating a space error between the target image and the reference image according to the positions of the target pixel points and the reference pixel points;

[0020] Based on the inverse rendering of differentiable rendering, calculating an optimized gradient of the model of the space error with respect to target model parameters of the model to be optimized;

[0021] Through a gradient descent algorithm, iteratively optimizing the target model parameters according to the optimized gradient of the model.

[0022] In one embodiment, the step of obtaining a model to be optimized and a reference model in a level of detail LoD model of oblique photography includes:

[0023] In the LoD model of the oblique photography, determining a target level at the next level below the highest precision level;

[0024] Taking the target level as a starting point, traversing the LoD model in a manner of gradually reducing the model precision, and using the currently accessed level during the traversal as the level to be optimized;

[0025] Determining a reference level of the level to be optimized according to a reference level interval;

[0026] Obtaining the reference model at the reference level and the model to be optimized at the level to be optimized.

[0027] In one embodiment, after the step of using the target level as a starting point to traverse the LoD model in a manner of gradually reducing the model accuracy and using the currently visited level during the traversal as the level to be optimized, the method further includes:

[0028] After completing the traversal of the LoD model, perform an increment action on the reference level interval;

[0029] When the reference level interval is less than the level interval threshold, jump to execute the step of using the target level as a starting point to traverse the LoD model in a manner of gradually reducing the model accuracy and using the currently visited level during the traversal as the level to be optimized and subsequent steps;

[0030] Alternatively, when the reference level interval is equal to or greater than the level interval threshold, perform a level reorganization on the LoD model to obtain the model optimization result of the oblique photography.

[0031] In one embodiment, the step of obtaining the reference model of the reference level and the model to be optimized of the level to be optimized includes:

[0032] According to a preset selection rule of nodes to be optimized, select the corresponding nodes to be optimized from the unoptimized nodes of the model to be optimized;

[0033] Based on the tree - like data structure of the LoD model, determine the reference nodes corresponding to the child nodes of the nodes to be optimized in the reference level;

[0034] Obtain the model to be optimized of the nodes to be optimized and the reference model of the reference nodes.

[0035] In one embodiment, the step of rendering the target scene and the reference scene through a virtual camera based on the photography parameters of the oblique photography to obtain a target image and a reference image includes:

[0036] Obtain the shooting angle of the oblique photography;

[0037] Based on the shooting angle, construct a target virtual camera array in the target scene and a reference virtual camera array in the reference scene;

[0038] Render the front image and the side image of the target model through the target virtual camera array, and use the front image and the side image as the target image;

[0039] Further, by means of the reference virtual camera array, front reference images and side reference images of the reference model are rendered, and the front reference images and the side reference images are used as the reference images.

[0040] In one embodiment, before the steps of obtaining the model to be optimized and the reference model in the level of detail LoD model of oblique photography, the method further includes:

[0041] Image data collected during the oblique photography is obtained, feature points in the image data are identified, and the positions of the feature points in the image data are determined;

[0042] Based on the positions of the feature points, the spatial positions corresponding to the pixel points in the image data are calculated through triangulation to generate point cloud data;

[0043] The point cloud data is converted into a triangular mesh model, and the image data is mapped into the triangular mesh model to generate an oblique photography model;

[0044] Through a decimation algorithm, LoD models corresponding to different levels of the oblique photography model are constructed based on different viewing distances, and the LoD models are associated with the viewing distances.

[0045] One or more technical solutions proposed in this application have at least the following technical effects:

[0046] In this application, in the level of detail LoD model of oblique photography, a model to be optimized with a lower precision and a reference model with a higher precision are selected and loaded into the target scene and the reference scene respectively. The spatial error between the target image rendered from the target scene and the reference image rendered from the reference scene is calculated through inverse rendering, and the model optimization gradient is calculated. Then, based on this spatial error, the model to be optimized is optimized and iterated, the spatial error between the low-level model and the high-level model of the LoD model is reduced, the utilization rate of the triangular meshes and modeling materials in the low-level model is improved, and the model accuracy of the low-level model is enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with this application and used together with the specification to explain the principles of this application.

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0049] Figure 1Schematic flowchart of the first embodiment of the method for optimizing the tilt photography LoD model based on differentiable rendering in this application;

[0050] Figure 2 Schematic flowchart of the simplified process involved in the method for optimizing the tilt photography LoD model based on differentiable rendering in this application;

[0051] Figure 3 Schematic flowchart of the second embodiment of the method for optimizing the tilt photography LoD model based on differentiable rendering in this application;

[0052] Figure 4 Schematic flowchart of the third embodiment of the method for optimizing the tilt photography LoD model based on differentiable rendering in this application;

[0053] Figure 5 Schematic flowchart of the fourth embodiment of the method for optimizing the tilt photography LoD model based on differentiable rendering in this application;

[0054] Figure 6 Schematic flowchart of the fifth embodiment of the method for optimizing the tilt photography LoD model based on differentiable rendering in this application;

[0055] Figure 7 Schematic diagram of the structure of the device for optimizing the tilt photography LoD model based on differentiable rendering in the hardware operating environment involved in the solution of the embodiment of this application.

[0056] The realization, functional features, and advantages of the purpose of this application will be further described with reference to the embodiments and the accompanying drawings. Specific Embodiments

[0057] It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application.

[0058] To better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings of the specification and specific embodiments.

[0059] The main solution of the embodiment of this application is: in the level of detail LoD model of tilt photography, obtain the model to be optimized and the reference model; load the model to be optimized into the target scene, and load the reference model into the reference scene; based on the photography parameters of the tilt photography, render the target scene and the reference scene through a virtual camera to obtain a target image and a reference image; calculate the spatial error between the target image and the reference image, and calculate the model optimization gradient corresponding to the spatial error through inverse rendering; based on the model optimization gradient, optimize the model to be optimized to obtain an optimized model.

[0060] In the related art, in oblique photography modeling, through the level of detail (LoD) technology, based on different viewing angles and viewing distances of the observer, the rendering fineness of the model is dynamically adjusted to reduce the number of triangular faces in rendering and improve the rendering efficiency. However, during the oblique photography modeling process, the model has a large amount of noise. At the same precision, the number of triangular faces, vertices, and material resolutions required for the oblique photography model are much larger than those of the manual model representing the same building. When the number of triangular faces of the LoD model of oblique photography is still larger than that of the manual model, the rendering result is more blurred compared to the manual model. This results in insufficient rendering precision of the LoD model of oblique photography.

[0061] In this application, in the LoD model of oblique photography, a to-be-optimized model with a lower precision and a reference model with a higher precision are selected and loaded into the target scene and the reference scene respectively. Then, the spatial error between the target image rendered from the target scene and the reference image rendered from the reference scene is calculated through inverse rendering, and the model optimization gradient is calculated. Thus, based on this spatial error, the to-be-optimized model is optimized and iterated, the spatial error between the low-level model and the high-level model of the LoD model is reduced, the utilization rate of the triangular mesh and the modeling material in the low-level model is improved, and the model precision of the low-level model is enhanced.

[0062] To better understand the above technical solution, the exemplary embodiments of this application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of this application and to be able to fully convey the scope of this application to those skilled in the art.

[0063] It should be noted that the execution subject of this embodiment can be a model optimization system, or a computing service device with data processing, network communication, and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of implementing the above functions, oblique photography LoD model optimization based on differentiable rendering, etc. This embodiment does not make specific limitations in this regard. Hereinafter, taking the model optimization system as an example, this embodiment and the following embodiments will be described.

[0064] Based on this, the embodiment of this application provides a method for optimizing an oblique photography LoD model based on differentiable rendering, referring to Figure 1 , Figure 1 is a schematic flowchart of the first embodiment of the method for optimizing an oblique photography LoD model based on differentiable rendering of this application.

[0065] In this embodiment, the method for optimizing an oblique photography LoD model based on differentiable rendering includes steps S10 to S50:

[0066] Step S10: In the oblique photography LoD model, obtain the model to be optimized and the reference model;

[0067] It should be noted that oblique photography is a technology for generating 3D models from aerial images. By mounting multiple sensors on the same flight platform, image data is collected simultaneously from vertical and multiple oblique angles to capture the side textures and 3D information of the ground objects. Based on multi-view image data, high-precision 3D models can be generated by combining methods such as image segmentation, feature extraction, feature matching, and 3D reconstruction. The Level of Detail (LoD) model is used to optimize the rendering efficiency and visual effects of 3D models. By creating multiple model versions with different precision levels, dynamic switching is performed according to the viewing distance between the observer and the model to balance the rendering quality and performance. In oblique photography modeling, the LoD model can manage the rendering of complex scenes, ensuring high-precision details are provided when observed closely, while using low-precision models at a long distance or with a wide field of view to reduce the computational load, thereby achieving efficient and smooth 3D scene browsing and interaction.

[0068] In this embodiment, the oblique photography includes at least multiple different levels of LoD models, and the model accuracy of the LoD model decreases gradually from high to low levels based on the level. The model optimization system can obtain the high-level model, that is, the high-precision model, as the reference level in the LoD model, and obtain the low-precision model in the lower level as the model to be optimized.

[0069] As an alternative implementation, the model optimization system can, based on the user instruction, select the model to be optimized corresponding to the target level in the LoD model, and select a high-precision model with an appropriate interval level based on the preset reference level interval as the reference model.

[0070] As another alternative implementation, the model optimization system can also optimize each level of the LoD model step by step in the order from high to low levels through traversal. The model optimization system takes the currently accessed level as the target level, and selects a reference level in the LoD level higher than the target level to obtain the model to be optimized at the target level and the reference model at the reference level. Among them, the reference level is the level that has been optimized, or the highest precision level of the LoD model.

[0071] Step S20: Load the model to be optimized into the target scene, and load the reference model into the reference scene;

[0072] Step S30: Based on the photography parameters of the oblique photography, render the target scene and the reference scene through a virtual camera to obtain a target image and a reference image;

[0073] In this embodiment, in the model optimization system, two identical independent scenarios are initialized and used as the target scenario and the reference scenario respectively. The model optimization system loads the model to be optimized into the target scenario and the reference model into the reference scenario respectively, and performs model rendering in a scenario-based manner. Among them, the target scenario and the reference scenario are rendered based on the same rendering parameters. The model optimization system can obtain the photography parameters of the oblique photography, and render the target scenario and the reference scenario in a manner imitating oblique photography at the same angle and position as the photography parameters of the virtual camera, generating the target image and the reference image. It can also render the target image and the reference image through the rendering camera based on the preset rendering angle.

[0074] As an optional implementation manner of image rendering, the model optimization system can render the target scenario and the reference scenario based on the same shooting angle as the oblique photography, so as to achieve the image rendering effect of imitating oblique photography and make the inverse rendering process based on the target image and the reference image in the subsequent steps more accurate.

[0075] Specifically, after the image rendering system obtains the shooting angle of the oblique photography, based on this shooting angle, it constructs a target virtual camera array in the target scenario and a reference virtual camera array in the reference scenario. Among them, the target virtual camera array and the reference virtual camera array are consistent in parameters such as the number, shooting angle, and shooting position, and are the same as the real camera array of the oblique photography. Through the target virtual camera array, the front image and the side image of the target model are rendered, and the front image and the side image are used as the target image. And, through the reference virtual camera array, the front reference image and the side reference image of the reference model are rendered, and the front reference image and the side reference image are used as the reference image. Among them, the vertical camera in the camera array samples the orthophoto, and the oblique camera samples the side and elevation images of the buildings in the scene.

[0076] In one example, the oblique photography collects pictures through a camera array composed of five cameras with different orientations, including 1 vertical camera pointing vertically to the ground to obtain the orthophoto of the ground, and 4 oblique cameras. Each camera has an inclination angle between 30 degrees and 45 degrees, and points to the front, back, left, and right respectively to shoot the side and elevation images of the building, and ensures that the image overlap rate is between 60% and 80%. In the aerial sampling strategy of imitating oblique photography, a virtual camera array composed of five cameras with different orientations is constructed in an arrangement similar to the camera array on the oblique photography flight platform. During the random image sampling process, the position of the camera array is randomly sampled within the specified height space range. At the same time, the height space sampling range increases with the improvement of the LoD level of detail to ensure that effectively rendered images are sampled.

[0077] In another example, the model optimization system can construct a virtual camera array consisting of 5 cameras, 1 vertical camera and 4 tilted cameras. The 4 tilted cameras are tilted at an angle of 45° forward, backward, left and right respectively. The vertical camera samples orthophotos, and the tilted cameras sample images of the sides and facades of the buildings in the scene. During the fitting process, the height of the camera array is sampled within the range of 100 - 300 meters, and the horizontal position of the camera array is sampled in the airspace directly above the scene. Based on the sampling results, the camera array is placed in both the reference scene and the target scene, and the differentiable renderer is given to sample five-direction images during the forward rendering process.

[0078] As another optional implementation of image rendering, due to the different rendering requirements of different scenes for the model, the model optimization system can also render the target scene and the reference scene from directions such as the horizontal plane or the vertical angle based on the preset shooting angles.

[0079] Furthermore, the model optimization system performs forward rendering on the target scene and the reference scene through differentiable rendering based on differentiable functions to generate the target image and the reference image.

[0080] It should be noted that differentiable rendering combines computer graphics and deep learning, designs the traditional rendering process as a differentiable function, and thus can calculate the gradient of the rendered image with respect to the scene parameters. Therefore, differentiable rendering can generate a two-dimensional image from a three-dimensional scene, and can also calculate the gradients of the image with respect to scene parameters such as geometry, material properties, and lighting conditions through inverse rendering. Differentiable rendering allows the rendering process to be embedded in an optimization or neural network pipeline, thereby solving inverse graphics problems, such as reconstructing a three-dimensional scene from a two-dimensional image, and by calculating the gradient, enabling optimization algorithms such as gradient descent to adjust the scene parameters to minimize the difference between the rendered image and the target image.

[0081] Optionally, since the model optimization system needs to generate cameras and light sources based on preset rules in the reference scene and the target scene to keep the light source and camera parameters consistent in the two scenes, the parallel light direction is randomly sampled within the range of 120° vertically downward. The sampled direction is used as the light source direction and placed in both the reference scene and the target scene simultaneously.

[0082] Step S40: Calculate the spatial error between the target image and the reference image, and calculate the model optimization gradient corresponding to the spatial error through inverse rendering;

[0083] Step S50: Optimize the model to be optimized based on the model optimization gradient to obtain an optimized model.

[0084] In this embodiment, the spatial error refers to the difference in position between the target pixel point and the reference pixel point, which reflects the geometric deviation between the target image and the reference image and can be achieved through the distance or similarity measure between pixel points. Inverse rendering is used to infer the physical properties of a scene from a 2D or 3D image and to achieve the 3D reconstruction of a 3D model, such as lighting, material, geometry, etc. It is the reverse operation of a forward rendering process such as differentiable rendering.

[0085] In one embodiment, the model optimization system determines the positions of the target pixel point and the reference pixel point by identifying the target pixel point of the target image and the reference pixel point mapped from the target pixel point in the reference image, so as to calculate the spatial error between the target image and the reference image. The model optimization system further calculates the gradient of the spatial error with respect to the model through the inverse rendering of differentiable rendering, thereby determining the step size and adjustment direction in the process of optimizing the model by the gradient descent algorithm, and thus optimizing the model parameters according to the gradient.

[0086] It should be noted that the gradient obtained by a single calculation only indicates the direction and rate of change of the parameter. How much the parameter specifically needs to be optimized in this direction, that is, the difference value, also needs to consider the learning rate. Gradient descent is a process of converging to the optimal result through multiple iterations.

[0087] Specifically, the model optimization system extracts features from the target image and the reference image. For example, it uses feature point detection algorithms such as SIFT (Scale-Invariant Feature Transform) and ORB (Oriented FAST and Rotated BRIEF) to identify the feature points in the image, and through the feature matching algorithm, it corresponds the target pixel points in the target image with the reference pixel points in the reference image one by one. The model optimization system quantifies the spatial error by calculating the Euclidean distance or photometric error between the target pixel point and the reference pixel point. For example, the mean-square error (MSE) or the structural similarity index (SSIM) is used to measure the difference between two images. For each matched pixel point pair, the model optimization system calculates its error value and aggregates all the error values into a global error metric.

[0088] Furthermore, the model optimization system performs differential processing on the rendering process based on differentiable rendering technology. The model optimization system represents the spatial error as a function of the scene parameters through the inverse rendering process and calculates the gradient of this function. Based on the differentiation of the rendering equation, the model optimization system can use, for example, the Monte Carlo method or path tracing technology to estimate the gradient. And the model optimization system updates the model parameters according to the calculated gradient using the gradient descent algorithm or other model optimization algorithms.

[0089] It should be noted that the gradient indicates the direction in which the spatial error increases most rapidly. In the gradient descent algorithm, by taking the negative direction of the gradient, the direction in which the spatial error decreases most rapidly can be found, thereby guiding the update of the model parameters. The magnitude or norm of the gradient reflects the sensitivity of the spatial error to changes in the model parameters. A larger gradient means that the spatial error is more sensitive to parameter changes, so a smaller step size needs to be taken when updating the parameters to avoid excessive adjustments. Through the gradient information, the gradient descent algorithm can search the parameter space and iteratively update the parameters to find the optimal parameters that minimize the spatial error, so as to determine the difference in model parameters.

[0090] Optionally, the gradient descent algorithm can perform parameter iteration based on the three-dimensional model of the target image and data such as building materials, or can perform parameter iteration based on different model regions. The model optimization system can find the local minimum of the spatial error function by iteratively updating the parameters. The differentiable rendering pipeline can also be alternated between differentiable raster rendering and differentiable ray tracing rendering, that is, rasterization rendering and ray tracing rendering.

[0091] Exemplarily, referring to Figure 2 , Figure 2 a schematic flowchart of a brief process of an optimization method for an oblique photography LoD model based on differentiable rendering is provided. As Figure 2 shown, after the model optimization system determines the model to be optimized and the reference model and loads them into the target scene and the reference scene respectively, it generates random parallel light parameters to simulate light and renders the target image and the reference image through a virtual camera array. The model optimization system can calculate the spatial error L between the target image and the reference image based on the pixel points of the target image and the reference image, and calculate the gradient of the spatial error with respect to the model parameters based on the parameter model formed by the spatial error:

[0092]

[0093] The model optimization system optimizes the model parameters through the gradient descent algorithm based on the reciprocal of this error.

[0094] In the embodiments of the present application, by selecting a model to be optimized with a lower precision and a reference model with a higher precision in the level of detail LoD model of oblique photography, loading them into the target scene and the reference scene respectively, calculating the spatial error between the target image rendered by the target scene and the reference image rendered by the reference scene through inverse rendering, and calculating the model optimization gradient, the model to be optimized is optimized and iterated based on this spatial error, the spatial error between the low-level model and the high-level model of the LoD model is reduced, the utilization rate of the triangular meshes and modeling materials in the low-level model is improved, and the model accuracy of the low-level model is enhanced.

[0095] Based on the same inventive concept, the present application also provides a second embodiment. Refer to Figure 3 , Figure 3 which is a schematic flowchart of the second embodiment of the optimization method for the tilt photography LoD model based on differentiable rendering in the present application.

[0096] In this embodiment, as described in step S50, after optimizing the model to be optimized based on the model optimization gradient to obtain an optimized model, steps S51 to S54 are further included:

[0097] Step S51: Calculate the target space error between the optimized model and the reference model;

[0098] Step S52: When the target space error is equal to or greater than the space error threshold, use the optimized model as the model to be optimized;

[0099] Step S53: Jump to execute the steps of loading the model to be optimized into the target scene and loading the reference model into the reference scene;

[0100] Step S54: Alternatively, when the target space error is less than the space error threshold, update the optimized model to the LoD model of the tilt photography.

[0101] In this embodiment, in the model optimization system, a space error threshold is set to determine whether the optimization result of the model to be optimized meets the expected optimization requirements. The model optimization system will further calculate the target space error between the optimized model after optimizing the model to be optimized and the reference model. When the target space error is equal to or greater than the space error threshold, it is determined that the optimization is not up to standard, and this optimized model is used as the model to be optimized to perform optimization again based on the reference model. When the target space error is less than the space error threshold, the optimized model is updated to the LoD model of the tilt photography.

[0102] Exemplarily, as Figure 2 shown, the model optimization system will converge the space error of the model to be optimized within the space error threshold in a cyclic manner, that is, the space error is less than the space error threshold. When the space error is still greater than the space error threshold, the model optimization system uses the model optimized in the current cycle as the model to be optimized, jumps to execute the step of loading the model to be optimized into the target scene and subsequent steps, and re-optimizes the model to be optimized to execute the optimization cycle of the model to be optimized until the space error converges within the space error threshold, and then the optimization of the model to be optimized is completed and the loop is exited.

[0103] In the embodiment of the present application, after the model to be optimized is optimized to obtain an optimized model, the optimization results of the optimized model and the reference model are verified. When the verification result shows that the spatial error is greater than the spatial error threshold, the optimized model is optimized again until the spatial error is less than the spatial error threshold, thereby improving the accuracy and effect of model optimization.

[0104] Since the system introduced in the second embodiment of the present application is the system adopted for implementing the method of the first embodiment of the present application, based on the method introduced in the first embodiment of the present application, those skilled in the art can understand the specific structure and variations of the system, so it will not be elaborated herein. Any system adopted by the method of the first embodiment of the present application falls within the scope of protection of the present application.

[0105] Based on the same inventive concept, the present application also provides a third embodiment. Refer to Figure 4 , Figure 4 which is a schematic flowchart of the third embodiment of the optimization method for the tilt photography LoD model based on differentiable rendering of the present application.

[0106] In this embodiment, the optimization method for the tilt photography LoD model based on differentiable rendering further includes steps S61 to S64:

[0107] Step S61: In the LoD model of the tilt photography, determine the target level of the level immediately below the highest precision level;

[0108] Step S62: Starting from the target level, traverse the LoD model in a manner of gradually reducing the model precision, and use the currently accessed level during the traversal as the level to be optimized;

[0109] Step S63: Determine the reference level of the level to be optimized according to the reference level interval;

[0110] Step S64: Obtain the reference model of the reference level and the model to be optimized of the level to be optimized.

[0111] It should be noted that in differentiable rendering, the higher the similarity between the reference model and the target model, the smaller the parameter change range of the gradient calculation of the optimization algorithm, making the gradient change direction more accurate and not easily falling into a local optimal solution. Among them, due to the relatively low loss function value and the smaller number of changing parameters, the speed at which the model fits into a reasonable range is also relatively high. Therefore, in the tilt photography LoD level, a model of an adjacent or similar LoD level with a higher similarity is used for fitting, thereby improving the convergence speed.

[0112] In this embodiment, the reference level interval is the number of levels separated between the target level to be optimized and the reference level of the higher level. The model optimization system optimizes the model of the lower level by using the model of the higher level in the neighboring LoD levels as the reference model.

[0113] Exemplarily, as Figure 2 shown, the image optimization system can determine the target level and / or the reference level based on the level offset, and complete the optimization of all levels with the increment action of the level offset after the optimization of the target level. After the image optimization system completes the optimization of the current target level, it increments the level offset, for example, by adding one for each optimization, and sets the target level to the next level.

[0114] Furthermore, based on the spatial region in the LoD model, the model optimization system further divides the model into different nodes and optimizes the model in units of nodes. Among them, based on the data structure of the LoD model, there is a corresponding mapping relationship between the models of different levels. The model optimization system can determine the nodes to be optimized corresponding to the model in the target level based on the reference nodes in the reference model, or determine the corresponding reference nodes in the reference level based on the nodes to be optimized in the target level.

[0115] In one implementation manner, to solve the problem that the data volume of the high-detail level LoD in the LoD levels is too large and not suitable for large-scale scene loading, the model optimization system can perform model fitting from micro to macro. The model optimization system first selects the fitting area in the micro scene where the perspective is close to the ground, loads the higher-precision level as the reference scene, determines the reference nodes in the reference scene, loads the neighboring lower-precision level as the target scene in another rendering scene, and determines the nodes to be optimized corresponding to the reference nodes.

[0116] In another implementation manner, the model optimization system can also select the corresponding nodes to be optimized from the unoptimized nodes of the model to be optimized according to the preset selection rules of the nodes to be optimized. Based on the tree-like data structure of the LoD model, in the reference level, determine the reference nodes corresponding to the child nodes of the node to be optimized, and obtain the model to be optimized of the node to be optimized and the reference model of the reference node.

[0117] Exemplarily, taking the 3D Tiles oblique photography modeling as an example, in 3D Tiles, the node data is organized by a quadtree, that is, each node corresponds to four child nodes, respectively representing the four quadrants of space, including the upper left, upper right, lower left, and lower right. That is, 1 hierarchical node corresponds to 4 hierarchical nodes of the upper level. Therefore, to ensure the optimization effect of the model, before performing the fitting of the next level, it is necessary to ensure that the 4 nodes in the corresponding level are completed with fitting. And so on, gradually complete the fitting of adjacent levels from the highest precision to the lowest precision.

[0118] In this application, the target level is optimized through the reference level with an adjacent level interval, which improves the optimization speed of the target level and avoids a large gap in the level interval between the target level and the reference level, resulting in an excessive amount of data processing during the optimization process of the target level and a long optimization time.

[0119] Since the system introduced in the third embodiment of this application is the system adopted for implementing the method in the first embodiment of this application, based on the method introduced in the first embodiment of this application, those skilled in the art can understand the specific structure and variations of the system, so it will not be elaborated here. Any system adopted by the method in the first embodiment of this application falls within the scope protected by this application.

[0120] Based on the same inventive concept, this application also provides a fourth embodiment, referring to Figure 5 , Figure 5 which is a schematic flowchart of the fourth embodiment of the method for optimizing the oblique photography LoD model based on differentiable rendering in this application.

[0121] In this embodiment, the method for optimizing the oblique photography LoD model based on differentiable rendering further includes steps S65 to S67:

[0122] Step S65: After completing the traversal of the LoD model, perform the increment action of the reference level interval;

[0123] In this embodiment, in addition to performing the fitting of model optimization through adjacent levels, the model optimization system can also perform similar levels and perform fitting in a progressive manner. After the model optimization system completes the optimization of each level of the LoD model based on the current reference level interval, it will perform the increment action of the reference level interval, that is, add one to the reference level interval.

[0124] Exemplarily, the initial value of the reference level interval is 0. After selecting the reference level and the target level, the number of levels between the target level and the reference level is 0, and they are adjacent levels to each other. When the reference level interval is 2, it means that there are two levels between the reference level and the target level.

[0125] Step S66: When the reference level interval is less than the level interval threshold, jump to execute the step of taking the target level as the starting point, traversing the LoD model in the way of gradually reducing the model accuracy, and taking the currently accessed level in the traversal process as the level to be optimized and subsequent steps;

[0126] Step S67: Alternatively, when the reference level interval is equal to or greater than the level interval threshold, perform level reorganization on the LoD model to obtain the optimized result of the oblique photography model.

[0127] In this embodiment, since the data results of the oblique photography node tree are organized, the growth of the interval will cause the geometric growth of the reference level corresponding nodes, resulting in the geometric growth of the data volume. Therefore, a reference level interval is set for the reference level interval, so as to avoid too many level nodes being accessed during the optimization process of the model to be optimized.

[0128] Exemplarily, in 3D Tiles oblique photography modeling, the model nodes are organized in a quadtree form. Therefore, the maximum value of the reference level interval can be selected as 3, that is, the level interval threshold is 4. When the reference level interval is 3, there are 256 reference nodes in the reference level, corresponding to each node to be optimized. When the reference level interval is 4, there are 1024 reference nodes in the reference level, corresponding to each node to be optimized. At this time, there are too many level nodes to be accessed by the reference nodes.

[0129] Specifically, as Figure 2 shown, when the reference level interval is less than the level interval threshold, or the reference level interval is equal to or less than the maximum value of the level interval, the model optimization system determines that there is an unoptimized reference level interval at this time, resets the level offset, and re-traverses the detail level model based on the incremented reference level interval to complete the optimization of all levels. When the reference level interval is greater than or equal to the level interval threshold, or the reference level interval is greater than the maximum value of the level interval, the model optimization system determines that there is no unoptimized reference level interval at this time, performs format restoration and level reorganization on the LoD model to obtain the optimized result of the LoD model of the oblique photography.

[0130] In the embodiment of the present application, by incrementing the reference level interval, the interval between the target level and the reference level is gradually expanded, so that the target level is optimized based on the adjacent reference level to ensure the optimization speed, and the optimization effect of the model is improved in a step-by-step optimization manner. When optimizing the target level in a step-by-step manner, compared with directly optimizing each level with the reference model of the highest level, the optimization speed of the model can be significantly improved, and at the same time, the repeated optimization of the optimized model details in the reference model of the highest level can be avoided.

[0131] Since the system introduced in the fourth embodiment of this application is the system adopted for implementing the method in the first embodiment of this application, based on the method introduced in the first embodiment of this application, those skilled in the art can understand the specific structure and variations of the system, so it will not be elaborated here. Any system adopted by the method in the first embodiment of this application falls within the scope of protection of this application.

[0132] Based on the same inventive concept, the present application also provides a fifth embodiment. Refer to Figure 6 , Figure 6 which is a schematic flowchart of the fifth embodiment of the method for optimizing the tilt photography LoD model based on differentiable rendering in the present application.

[0133] In this embodiment, the method for optimizing the tilt photography LoD model based on differentiable rendering further includes steps S01 to S04:

[0134] Step S01: Obtain the image data collected during the tilt photography process, identify the feature points in the image data, and determine the positions of the feature points in the image data;

[0135] Step S02: Based on the positions of the feature points, calculate the spatial positions corresponding to the pixel points in the image data through triangulation to generate point cloud data;

[0136] Step S03: Convert the point cloud data into a triangular mesh model, and map the image data to the triangular mesh model to generate a tilt photography model;

[0137] Step S04: Through a decimation algorithm, based on different viewing distances, construct the LoD models of the corresponding levels of the tilt photography model, and associate the LoD models with the viewing distances.

[0138] In this embodiment, after obtaining the image data through tilt photography for the tilt photography LoD model, the positions of the feature points corresponding to the feature points in the recognized image can be based on, and through triangulation, calculate the spatial position corresponding to each pixel point in the image, thereby generating point cloud data. Based on this point cloud data, a corresponding triangular mesh model can be constructed through 3D meshes and building material data. By mapping the captured image data to the corresponding positions of the triangular mesh model, a tilt photography model is generated. Further, to divide the tilt photography model into models with different levels of detail through the LoD algorithm, the LoD algorithm constructs the LoD models of the corresponding levels of the tilt photography model through a decimation algorithm based on different viewing distances, and associates the LoD models with the viewing distances.

[0139] Optionally, to implement the tilt photography modeling process and the LoD model generation process and be compatible with the model optimization system, after the model optimization system obtains the LoD model, it is necessary to perform format conversion on the triangular mesh data and material data in the model. For example, convert the triangular mesh in each node conversion of the grouped tilt photography LoD level to the obj format, and convert the corresponding material to mtl. Correspondingly, after all LoD models are optimized and before the LoD models are recombined, the model optimization system will also restore the format of the model data.

[0140] Since the system introduced in Embodiment 5 of this application is the system adopted to implement the method of Embodiment 1 of this application, based on the method introduced in Embodiment 1 of this application, those skilled in the art can understand the specific structure and variations of the system, so it will not be elaborated here. Any system adopted by the method of Embodiment 1 of this application falls within the scope of protection of this application.

[0141] This application provides a tilt photography LoD model optimization device based on differentiable rendering. The device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the tilt photography LoD model optimization method based on differentiable rendering in Embodiment 1 above.

[0142] Next, refer to Figure 7 , which shows a schematic structural diagram of a tilt photography LoD model optimization device suitable for implementing the embodiments of this application. The tilt photography LoD model optimization device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The tilt photography LoD model optimization device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of this application.

[0143] As Figure 7As shown, the device for optimizing the tilt photography LoD model based on differentiable rendering may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 into the random access memory (RAM: Random Access Memory) 1004. In the random access memory 1004, various programs and data required for the operation of the device for optimizing the tilt photography LoD model based on differentiable rendering are also stored. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the device for optimizing the tilt photography LoD model based on differentiable rendering to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows a device for optimizing the tilt photography LoD model based on differentiable rendering with various systems, it should be understood that it is not required to implement or have all the shown systems. Instead, more or fewer systems may be implemented or had.

[0144] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device, or installed from the storage device 1003, or installed from the read-only memory 1002. When the computer program is executed by the processing device 1001, the above functions defined in the methods of the embodiments disclosed in the present application are executed.

[0145] The device for optimizing the oblique photography LoD model based on differentiable rendering provided by this application adopts the method for optimizing the oblique photography LoD model based on differentiable rendering in the above-mentioned embodiment, and can solve the technical problem of insufficient rendering accuracy of the oblique photography LoD model. Compared with the prior art, the beneficial effects of the device for optimizing the oblique photography LoD model based on differentiable rendering provided by this application are the same as those of the method for optimizing the oblique photography LoD model based on differentiable rendering provided by the above-mentioned embodiment, and other technical features in the device for optimizing the oblique photography LoD model based on differentiable rendering are the same as the features disclosed in the method of the previous embodiment, and will not be elaborated here.

[0146] It should be understood that each part disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0147] As mentioned above, the above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or substitutions, which should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

[0148] This application provides a computer-readable storage medium with computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the method for optimizing the oblique photography LoD model based on differentiable rendering in the above-mentioned embodiment.

[0149] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, device, or component. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination of the above.

[0150] The above computer-readable storage medium can be included in the differential rendering-based oblique photography LoD model optimization device; or can exist independently without being assembled into the differential rendering-based oblique photography LoD model optimization device.

[0151] The above computer-readable storage medium carries one or more programs. When the above one or more programs are executed by the differential rendering-based oblique photography LoD model optimization device, the differential rendering-based oblique photography LoD model optimization device is caused to: in the detail level LoD model of oblique photography, obtain the model to be optimized and the reference model; load the model to be optimized into the target scene and load the reference model into the reference scene; based on the photography parameters of the oblique photography, render the target scene and the reference scene through a virtual camera to obtain a target image and a reference image; calculate the spatial error between the target image and the reference image, and calculate the model optimization gradient corresponding to the spatial error through inverse rendering; and optimize the model to be optimized based on the model optimization gradient to obtain an optimized model.

[0152] Computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The above-mentioned programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, by connecting through an Internet service provider via the Internet).

[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0154] The modules described in the embodiments of this application can be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation on the unit itself in some cases.

[0155] The readable storage medium provided in this application is a computer-readable storage medium. The computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for performing the above-mentioned method for optimizing the tilt photography LoD model based on differentiable rendering, which can solve the technical problem of insufficient rendering accuracy of the tilt photography LoD model. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the method for optimizing the tilt photography LoD model based on differentiable rendering provided in the above embodiments, and will not be elaborated here.

[0156] The above are only some embodiments of the present application, and thus do not limit the patent scope of the present application. Any equivalent structural transformation made under the technical concept of the present application by using the content of the specification and drawings of the present application, or any direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A method for optimizing the LoD model of oblique photography based on differentiable rendering, characterized in that: The method comprises the following steps: In the oblique photography LoD model, a model to be optimized in a target level and a reference model in a reference level are obtained, wherein the reference level is a level that has been optimized or a level with the highest accuracy of the LoD model, the LoD level of the reference level is higher than the target level, and the LoD model gradually reduces the model accuracy from high to low based on the level; Loading the model to be optimized into a target scene, and loading the reference model into a reference scene; Based on the photography parameters of the oblique photography, the target scene and the reference scene are rendered by a virtual camera based on the same rendering parameters to obtain a target image and a reference image; Calculating a spatial error between the target image and the reference image, and calculating a model optimization gradient corresponding to the spatial error through inverse rendering; Based on the model optimization gradient, the model to be optimized is optimized to obtain an optimized model.

2. The method according to claim 1, characterized in that After the step of optimizing the model to be optimized based on the model optimization gradient, the method further includes: Calculating a target space error between the optimization model and a reference model; When the target spatial error is equal to or greater than a spatial error threshold, taking the optimization model as the model to be optimized; Jump to the step of loading the model to be optimized into the target scene, and loading the reference model into the reference scene; Alternatively, when the target spatial error is less than the spatial error threshold, the optimization model is updated to the LoD model of the oblique photography.

3. The method according to claim 1, characterized in that The step of calculating the spatial error between the target image and the reference image, and calculating the model optimization gradient corresponding to the spatial error by inverse rendering comprises: Identify target pixels of the target image and map them to reference pixels in the reference image; Calculating a spatial error between the target image and the reference image according to the target pixel position and the reference pixel position; Based on the inverse rendering of the differentiable rendering, calculating the model optimization gradient of the spatial error relative to the target model parameters of the model to be optimized; The target model parameters are iteratively optimized by using a gradient descent algorithm and optimizing the gradient according to the model.

4. The method according to claim 1, characterized in that In the oblique photography LoD model, the step of obtaining the model to be optimized in the target layer and the reference model in the reference layer includes: In the LoD model of the oblique photography, determining the target level of a level below the highest accuracy level; Taking the target level as the starting point, traversing the LoD model in a manner of gradually reducing the model accuracy, and taking the current access level in the traversal process as the level to be optimized; Determining the reference level of the level to be optimized according to the reference level interval; The reference model of the reference level and the model to be optimized of the level to be optimized are obtained.

5. The method according to claim 4, characterized in that After the step of taking the target level as the starting point, traversing the LoD model in a manner of gradually reducing the model accuracy, and taking the current access level in the traversal process as the level to be optimized, the method further includes: After completing the traversal of the LoD model, performing a self-increment action of the reference level interval; When the reference level interval is less than the level interval threshold, jump to execute the step and subsequent steps of taking the target level as the starting point, traversing the LoD model in a manner of gradually reducing the model accuracy, and taking the current access level in the traversal process as the level to be optimized; Alternatively, when the reference level interval is equal to or greater than the level interval threshold, the LoD model is hierarchically reorganized to obtain a model optimization result of the oblique photography.

6. The method according to claim 4, characterized in that The step of acquiring the reference model of the reference level and the model to be optimized of the level to be optimized comprises: According to a preset selection rule of the node to be optimized, selecting the corresponding node to be optimized from the unoptimized nodes of the model to be optimized; Based on the tree data structure of the LoD model, determining, in the reference level, a reference node corresponding to a child node of the node to be optimized; The model to be optimized of the node to be optimized and the reference model of the reference node are obtained.

7. The method according to claim 1, characterized in that The step of rendering the target scene and the reference scene based on the same rendering parameters by a virtual camera based on the photography parameters of the oblique photography to obtain the target image and the reference image comprises: Acquire the shooting angle of the oblique photography; Based on the shooting angles, constructing a target virtual camera array in the target scene and a reference virtual camera array in the reference scene; Rendering a front image and a side image of the model to be optimized through the target virtual camera array, and using the front image and the side image as the target image; And, rendering a front reference image and a side reference image of the reference model through the reference virtual camera array, and using the front reference image and the side reference image as the reference images.

8. The method according to claim 1, characterized in that Before the step of obtaining the model to be optimized in the target layer and the reference model in the reference layer in the oblique photography LoD model, the method further includes: Acquire image data collected during the oblique photography process, identify feature points in the image data, and determine the positions of the feature points in the image data; Based on the position of the feature point, the spatial position corresponding to the pixel point in the image data is calculated by triangulation to generate point cloud data; Converting the point cloud data into a triangular mesh model, and mapping the image data into the triangular mesh model to generate an oblique photography model; By using a surface reduction algorithm, based on different viewing distances, the LoD model of the corresponding level of the oblique photography model is constructed, and the LoD model is associated with the viewing distance.

Images