Oblique photography LoD model optimization method based on differential rendering

Abstract

The invention discloses an oblique photography LoD model optimization method based on differential rendering, and belongs to the technical field of data processing. The method comprises the steps of loading a to-be-optimized model in an oblique photography LoD model into a target scene, loading a reference model into a reference scene, rendering the target scene and the reference scene through a virtual camera based on photography parameters of oblique photography to obtain a target image and a reference image, and outputting the target image and the reference image. And calculating a spatial error between the target image and the reference image, calculating a model optimization gradient corresponding to the spatial error through inverse rendering, and optimizing the to-be-optimized model based on the model optimization gradient to obtain an optimized model. In the LoD model after oblique photography modeling, based on a high-level high-precision model as a reference, optimization iteration is carried out on a low-level low-precision model, the space error between the low-level model and the high-level model is reduced, and therefore the model precision of the low-level model is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an oblique photography LoD model optimization method based on differentiable rendering. Background Art

[0002] Oblique photography modeling uses sensors carried by drones to collect images and record aerial measurement information from vertical and four oblique angles, and automatically models images using methods such as image segmentation and feature extraction. In related technologies, oblique photography modeling uses the level of detail (LoD) technology to dynamically adjust the rendering precision of the model based on the different perspectives and observation distances of the observer, so as to reduce the number of rendered triangles and improve rendering efficiency.

[0003] However, due to the high noise in the modeling process of oblique photography, the number of triangles, vertices, and material resolution required for the oblique photography model is much greater than that of the manual model expressing the same building at the same accuracy. When the number of triangles of the oblique photography level of detail model is still greater than that of the manual model, the rendering result is more blurred than that of the manual model. This leads to insufficient rendering accuracy of the oblique photography LoD model.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide an oblique photography LoD model optimization method based on differentiable rendering, aiming to solve the technical problem of insufficient rendering accuracy of the oblique photography LoD model.

[0006] To achieve the above objectives, the present application provides a method for optimizing the LoD model of oblique photography based on differentiable rendering, wherein the method comprises the following steps: In the detail level LoD model of oblique photography, a model to be optimized and a reference model are obtained; Loading the model to be optimized into a target scene, and loading the reference model into a reference scene; Based on the photography parameters of the oblique photography, the target scene and the reference scene are rendered by a virtual camera to obtain a target image and a reference image; Calculating a spatial error between the target image and the reference image, and calculating a model optimization gradient corresponding to the spatial error through inverse rendering; Based on the model optimization gradient, the model to be optimized is optimized to obtain an optimized model.

[0007] In one embodiment, after the step of optimizing the model to be optimized based on the model optimization gradient, the method further includes: Calculating a target space error between the optimization model and a reference model; When the target spatial error is equal to or greater than a spatial error threshold, taking the optimization model as the model to be optimized; Jump to the step of loading the model to be optimized into the target scene, and loading the reference model into the reference scene; Alternatively, when the target spatial error is less than the spatial error threshold, the optimization model is updated to the LoD model of the oblique photography.

[0008] In one embodiment, the step of calculating the spatial error between the target image and the reference image, and calculating the model optimization gradient corresponding to the spatial error by inverse rendering includes: Identify target pixels of the target image and map them to reference pixels in the reference image; Calculating a spatial error between the target image and the reference image according to the target pixel position and the reference pixel position; Based on the inverse rendering of the differentiable rendering, calculating the model optimization gradient of the spatial error relative to the target model parameters of the model to be optimized; The target model parameters are iteratively optimized by using a gradient descent algorithm and optimizing the gradient according to the model.

[0009] In one embodiment, in the level of detail LoD model of oblique photography, the step of obtaining the model to be optimized and the reference model includes: In the LoD model of the oblique photography, determining a target level of a level below the highest accuracy level; Taking the target level as the starting point, traversing the LoD model in a manner of gradually reducing the model accuracy, and taking the current access level in the traversal process as the level to be optimized; Determining the reference level of the level to be optimized according to the reference level interval; The reference model of the reference level and the model to be optimized of the level to be optimized are obtained.

[0010] In one embodiment, after the step of taking the target level as the starting point, traversing the LoD model in a manner of gradually reducing the model accuracy, and taking the current access level in the traversal process as the level to be optimized, the step further includes: After completing the traversal of the LoD model, performing a self-increment action of the reference level interval; When the reference level interval is less than the level interval threshold, jump to execute the step and subsequent steps of taking the target level as the starting point, traversing the LoD model in a manner of gradually reducing the model accuracy, and taking the current access level in the traversal process as the level to be optimized; Alternatively, when the reference level interval is equal to or greater than the level interval threshold, the LoD model is hierarchically reorganized to obtain a model optimization result of the oblique photography.

[0011] In one embodiment, the step of acquiring the reference model of the reference level and the model to be optimized of the level to be optimized includes: According to a preset selection rule of the node to be optimized, selecting the corresponding node to be optimized from the unoptimized nodes of the model to be optimized; Based on the tree data structure of the LoD model, determining, in the reference level, a reference node corresponding to a child node of the node to be optimized; The model to be optimized of the node to be optimized and the reference model of the reference node are obtained.

[0012] In one embodiment, the step of rendering the target scene and the reference scene by a virtual camera based on the photography parameters of the oblique photography to obtain the target image and the reference image includes: Acquire the shooting angle of the oblique photography; Based on the shooting angles, constructing a target virtual camera array in the target scene and a reference virtual camera array in the reference scene; Rendering a front image and a side image of the target model through the target virtual camera array, and using the front image and the side image as the target image; And, rendering a front reference image and a side reference image of the reference model through the reference virtual camera array, and using the front reference image and the side reference image as the reference images.

[0013] In one embodiment, before the step of obtaining the model to be optimized and the reference model in the detail level LoD model of the oblique photography, the step further includes: Acquire image data collected during the oblique photography process, identify feature points in the image data, and determine the positions of the feature points in the image data; Based on the position of the feature point, the spatial position corresponding to the pixel point in the image data is calculated by triangulation to generate point cloud data; Converting the point cloud data into a triangular mesh model, and mapping the image data into the triangular mesh model to generate an oblique photography model; By using a surface reduction algorithm, based on different viewing distances, the LoD model of the corresponding level of the oblique photography model is constructed, and the LoD model is associated with the viewing distance.

[0014] One or more technical solutions proposed in this application have at least the following technical effects: The present application selects a lower-precision model to be optimized and a higher-precision reference model in the detail level LoD model of oblique photography, loads them into the target scene and the reference scene respectively, calculates the spatial error between the target image rendered by the target scene and the reference image rendered by the reference scene through inverse rendering, and calculates the model optimization gradient, thereby optimizing and iterating the model to be optimized based on the spatial error, reducing the spatial error between the low-level model and the high-level model of the LoD model, improving the utilization rate of the triangular mesh and modeling materials in the low-level model, and improving the model accuracy of the low-level model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0017] Figure 1 This is a flow chart of the first embodiment of the method for optimizing the LoD model of oblique photography based on differentiable rendering of the present application; Figure 2 This is a simplified flowchart of the method for optimizing the oblique photography LoD model based on differentiable rendering in this application; Figure 3 This is a flow chart of a second embodiment of the method for optimizing the LoD model of oblique photography based on differentiable rendering of the present application; Figure 4 This is a flow chart of a third embodiment of the method for optimizing the LoD model of oblique photography based on differentiable rendering of the present application; Figure 5 This is a flow chart of a fourth embodiment of the method for optimizing the LoD model of oblique photography based on differentiable rendering of the present application; Figure 6 This is a flowchart of a fifth embodiment of the method for optimizing the LoD model of oblique photography based on differentiable rendering of the present application; Figure 7It is a structural diagram of an oblique photography LoD model optimization device based on differentiable rendering in the hardware operating environment involved in the embodiment of the present application.

[0018] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0019] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0020] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0021] The main solution of the embodiment of the present application is: in the detail level LoD model of oblique photography, obtain the model to be optimized and the reference model; load the model to be optimized into the target scene, and load the reference model into the reference scene; based on the photographic parameters of the oblique photography, render the target scene and the reference scene through a virtual camera to obtain a target image and a reference image; calculate the spatial error between the target image and the reference image, and calculate the model optimization gradient corresponding to the spatial error through inverse rendering; based on the model optimization gradient, optimize the model to be optimized to obtain an optimized model.

[0022] In related technologies, oblique photography modeling uses the level of detail (LoD) technology to dynamically adjust the rendering precision of the model based on the different viewing angles and observation distances of the observer to reduce the number of rendered triangles and improve rendering efficiency. However, due to the large model noise during the oblique photography modeling process, the number of triangles, vertices, and material resolution required for the oblique photography model is much greater than that of the manual model that expresses the same building at the same accuracy. When the number of triangles of the oblique photography level of detail model is still greater than that of the manual model, the rendering result is more blurred than the manual model. This leads to insufficient rendering accuracy of the oblique photography LoD model.

[0023] The present application selects a lower-precision model to be optimized and a higher-precision reference model in the LoD model of oblique photography, loads them into the target scene and the reference scene respectively, calculates the spatial error between the target image rendered by the target scene and the reference image rendered by the reference scene through inverse rendering, and calculates the model optimization gradient, thereby optimizing and iterating the model to be optimized based on the spatial error, reducing the spatial error between the low-level model and the high-level model of the LoD model, improving the utilization rate of the triangular mesh and modeling materials in the low-level model, and improving the model accuracy of the low-level model.

[0024] In order to better understand the above technical solution, exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0025] It should be noted that the execution subject of this embodiment can be a model optimization system, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, oblique photography LoD model optimization based on differentiable rendering, etc., which is not specifically limited in this embodiment. The following takes the model optimization system as an example to illustrate this embodiment and the following embodiments.

[0026] Based on this, the embodiment of the present application provides a method for optimizing the oblique photography LoD model based on differentiable rendering, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the oblique photography LoD model optimization method based on differentiable rendering of the present application.

[0027] In this embodiment, the oblique photography LoD model optimization method based on differentiable rendering includes steps S10 to S50: Step S10: in the oblique photography LoD model, obtaining the model to be optimized and the reference model; It should be noted that oblique photography is a technology that generates three-dimensional models from aerial images. By carrying multiple sensors on the same flight platform, image data is collected simultaneously from vertical and multiple oblique angles to capture the side texture and three-dimensional information of the objects. Based on multi-view image data, high-precision three-dimensional models can be generated by combining image segmentation, feature extraction, feature matching, three-dimensional reconstruction and other methods. The Level of Detail (LoD) model is used to optimize the rendering efficiency and visual effects of three-dimensional models. By creating multiple model versions with different accuracy levels, they are dynamically switched according to the viewing distance between the observer and the model to balance the rendering quality and performance. In oblique photography modeling, the LoD model can manage the rendering of complex scenes, ensuring high-precision details when observed at close range, and using low-precision models at long distances or with a wide field of view to reduce the computational load, thereby achieving efficient and smooth three-dimensional scene browsing and interaction.

[0028] In this embodiment, oblique photography includes at least a plurality of different levels of LoD models, and the LoD model gradually reduces the model accuracy from high to low based on the level. The model optimization system can obtain a high-level model, i.e., a high-precision model, as a reference level in the LoD model, and obtain a low-precision model in a lower level as a model to be optimized.

[0029] As an optional implementation, the model optimization system can select the model to be optimized corresponding to the target level in the LoD model based on user instructions, and select a high-precision model with an appropriate level interval as a reference model based on a preset reference level interval.

[0030] As another optional implementation, the model optimization system can also optimize the LoD model of each level step by step by traversing the level from high to low. The model optimization system takes the current access level as the target level, selects a reference level in the LoD level higher than the target level, and obtains the model to be optimized in the target level and the reference model in the reference level. The reference level is the level that has been optimized, or the highest precision level of the LoD model.

[0031] Step S20: loading the model to be optimized into the target scene, and loading the reference model into the reference scene; Step S30: Based on the photography parameters of the oblique photography, the target scene and the reference scene are rendered by a virtual camera to obtain a target image and a reference image; In this embodiment, in the model optimization system, two identical independent scenes are initialized, which are used as the target scene and the reference scene respectively. The model optimization system loads the model to be optimized into the target scene and the reference model into the reference scene respectively, and renders the model in a scene-based manner. Among them, the target scene and the reference scene are rendered based on the same rendering parameters. The model optimization system can obtain the photographic parameters of oblique photography, and render the target scene and the reference scene in a manner of simulating oblique photography through a virtual camera with the same angle and position as the photographic parameters to generate a target image and a reference image. It can also render the target image and the reference image through a rendering camera based on a preset rendering angle.

[0032] As an optional implementation method for image rendering, the model optimization system can render the target scene and the reference scene based on the same shooting angle of oblique photography, thereby achieving an image rendering effect that simulates oblique photography, making the inverse rendering process based on the target image and the reference image in subsequent steps more accurate.

[0033] Specifically, after acquiring the shooting angle of the oblique photography, the image rendering system constructs a target virtual camera array in the target scene and a reference virtual camera array in the reference scene based on the shooting angle. The target virtual camera array is consistent with the reference virtual camera array in terms of parameters such as number, shooting angle, and shooting position, and is the same as the real camera array of the oblique photography. The front image and the side image of the target model are rendered through the target virtual camera array, and the front image and the side image are used as the target image. And, the front reference image and the side reference image of the reference model are rendered through the reference virtual camera array, and the front reference image and the side reference image are used as reference images. The vertical camera in the camera array samples the orthophoto, and the oblique camera samples the scene to shoot the side and facade images of the building.

[0034] In one example, oblique photography collects images through a camera array consisting of five cameras in different orientations, including one vertical camera pointing vertically to the ground to obtain an orthophoto of the ground, and four oblique cameras, each with an inclination angle of 30 to 45 degrees, pointing to the front, back, left, and right to capture the side and facade images of the building, and ensuring that the image overlap is between 60% and 80%. In the pseudo-oblique photography aerial sampling strategy, a virtual camera array consisting of five different orientations is constructed, with a consistent arrangement similar to the camera array on the oblique photography flight platform. During the random image sampling process, the camera array position is randomly sampled within the specified height space range. At the same time, the height space sampling range increases with the increase of the LoD level of detail to ensure that the image is sampled to be effectively rendered.

[0035] In another example, the model optimization system can construct a virtual camera array consisting of 5 cameras, 1 vertical camera, and 4 oblique cameras. The 4 oblique cameras are tilted at 45° to the front, back, left, and right. The vertical camera samples orthophotos, and the oblique camera samples scenes to take side and facade images of buildings. During the fitting process, the camera array height is sampled in the range of 100-300 meters, and the horizontal position of the camera array is sampled in the airspace directly above the scene. Based on the sampling results, the camera array is placed in the reference scene and the target scene at the same time, and the differentiable renderer is given a forward rendering process to sample five directional images.

[0036] As another optional implementation method of image rendering, since different scenes have different rendering requirements for models, the model optimization system can also render the target scene and the reference scene based on a preset shooting angle, from a horizontal plane or a vertical angle.

[0037] Furthermore, the model optimization system performs forward rendering on the target scene and the reference scene by means of differentiable rendering based on a differentiable function to generate a target image and a reference image.

[0038] It should be noted that Differentiable Rendering combines computer graphics and deep learning, and designs the traditional rendering process as a differentiable function, so that the gradient of the rendered image relative to the scene parameters can be calculated. Therefore, differentiable rendering can generate a two-dimensional image from a three-dimensional scene, and can also calculate the gradient of the image with respect to scene parameters such as geometry, material properties, and lighting conditions through inverse rendering. Differentiable rendering allows the rendering process to be embedded in an optimization or neural network pipeline to solve inverse graphics problems, such as reconstructing a three-dimensional scene from a two-dimensional image, and by calculating the gradient, optimization algorithms such as gradient descent can adjust the scene parameters to minimize the difference between the rendered image and the target image.

[0039] Optionally, since the model optimization system needs to generate cameras and light sources based on preset rules in the reference scene and the target scene, in order to keep the light source and camera parameters consistent in the two scenes, the parallel light direction is randomly sampled within a vertical downward range of 120°. The sampled direction is used as the light source direction and placed in both the reference scene and the target scene.

[0040] Step S40: calculating a spatial error between the target image and the reference image, and calculating a model optimization gradient corresponding to the spatial error by inverse rendering; Step S50: Based on the model optimization gradient, the model to be optimized is optimized to obtain an optimized model.

[0041] In this embodiment, the spatial error refers to the difference in position between the target pixel and the reference pixel, reflecting the geometric deviation between the target image and the reference image, which can be achieved by measuring the distance or similarity between the pixels. Inverse rendering is used to infer the physical properties of the scene from the 2D or 3D image to achieve 3D reconstruction of the 3D model, such as lighting, material, geometric shape, etc. It is the reverse operation of the forward rendering process such as differentiable rendering.

[0042] In one embodiment, the model optimization system determines the target pixel position and the reference pixel position by identifying the target pixel of the target image and the reference pixel mapped to the target pixel in the reference image to calculate the spatial error between the target image and the reference image. The model optimization system further calculates the gradient of the spatial error relative to the model through inverse rendering of differentiable rendering, thereby determining the step size and adjustment direction in the process of optimizing the model by the gradient descent algorithm, thereby optimizing the model parameters according to the gradient.

[0043] It should be noted that the gradient obtained from a single calculation can only indicate the direction and rate at which the parameters need to change. The specific amount by which the parameters need to be optimized in that direction, i.e. the difference, also needs to be considered in terms of the learning rate. Gradient descent is a process that converges to the optimal result through multiple iterations.

[0044] Specifically, the model optimization system extracts features from the target image and the reference image, for example, using feature point detection algorithms such as SIFT (Scale-Invariant Feature Transform) and ORB (Oriented FAST and Rotated BRIEF) to identify feature points in the image, and uses a feature matching algorithm to match target pixels in the target image with reference pixels in the reference image one by one. The model optimization system quantifies the spatial error by calculating the Euclidean distance or photometric error between the target pixel and the reference pixel. For example, the mean square error (MSE) or structural similarity index (SSIM) is used to measure the difference between two images. For each matching pixel pair, the model optimization system calculates its error value and aggregates all error values ​​into a global error index.

[0045] Furthermore, the model optimization system performs differentiation on the rendering process based on the differentiable rendering technology. The model optimization system expresses the spatial error as a function of the scene parameters through the inverse rendering process and calculates the gradient of the function. Based on the differentiation of the rendering equation, the model optimization system can use, for example, the Monte Carlo method or path tracing technology to estimate the gradient. The model optimization system updates the model parameters based on the calculated gradient using the gradient descent algorithm or other model optimization algorithms.

[0046] It should be noted that the gradient indicates the direction in which the spatial error increases fastest. In the gradient descent algorithm, by taking the negative direction of the gradient, the direction in which the spatial error decreases fastest can be found, thereby guiding the update of the model parameters. The size or modulus of the gradient reflects the sensitivity of the spatial error to changes in the model parameters. A larger gradient means that the spatial error is more sensitive to parameter changes, so a smaller step size needs to be taken when updating the parameters to avoid excessive adjustments. Through the gradient information, the gradient descent algorithm can search the parameter space and find the optimal parameters that minimize the spatial error by iteratively updating the parameters to determine the model parameter difference.

[0047] Optionally, the gradient descent algorithm can perform parameter iteration based on the three-dimensional model and building materials of the target image, or it can perform parameter iteration based on different model regions. The model optimization system can find the local minimum of the spatial error function by iteratively updating the parameters. The differentiable rendering pipeline can also be replaced between differentiable raster rendering and differentiable ray tracing rendering, that is, rasterization rendering and ray tracing rendering.

[0048] For example, refer to Figure 2 , Figure 2This paper provides a brief flowchart of the optimization method of the oblique photography LoD model based on differentiable rendering. Figure 2 As shown, after the model to be optimized and the reference model are determined and loaded into the target scene and the reference scene respectively, the model optimization system generates random parallel illumination parameters to simulate illumination, and renders the target image and the reference image through a virtual camera array. The model optimization system can calculate the spatial error L between the target image and the reference image based on the pixel points of the target image and the reference image, and calculate the gradient of the spatial error relative to the model parameters based on the parameter model formed by the spatial error:

[0049] The model optimization system optimizes the model parameters based on the inverse of the error through a gradient descent algorithm.

[0050] The embodiment of the present application selects a lower-precision model to be optimized and a higher-precision reference model in the detail level LoD model of oblique photography, loads them into the target scene and the reference scene respectively, calculates the spatial error between the target image rendered by the target scene and the reference image rendered by the reference scene through inverse rendering, and calculates the model optimization gradient, thereby optimizing and iterating the model to be optimized based on the spatial error, reducing the spatial error between the low-level model and the high-level model of the LoD model, improving the utilization rate of the triangular mesh and modeling materials in the low-level model, and improving the model accuracy of the low-level model.

[0051] Based on the same inventive concept, the present application also provides a second embodiment, referring to Figure 3 , Figure 3 This is a flow chart of the second embodiment of the oblique photography LoD model optimization method based on differentiable rendering of the present application.

[0052] In this embodiment, as described in step S50, the model to be optimized is optimized based on the model optimization gradient, and after the optimized model is obtained, steps S51 to S54 are also included: Step S51: Calculating the target space error between the optimization model and the reference model; Step S52: when the target spatial error is equal to or greater than a spatial error threshold, taking the optimization model as the model to be optimized; Step S53: Jump to the step of loading the model to be optimized into the target scene, and loading the reference model into the reference scene; Step S54: Alternatively, when the target spatial error is less than the spatial error threshold, the optimization model is updated to the LoD model of the oblique photography.

[0053] In this embodiment, a spatial error threshold is set in the model optimization system to determine whether the optimization result of the model to be optimized meets the expected optimization requirements. The model optimization system will further calculate the target spatial error between the optimized model after the optimization of the model to be optimized and the reference model. When the target spatial error is equal to or greater than the spatial error threshold, it is determined that the optimization does not meet the standard, and the optimized model is used as the model to be optimized and optimized again based on the reference model. When the target spatial error is less than the spatial error threshold, the optimized model is updated to the LoD model of oblique photography.

[0054] For example, Figure 2 As shown, the model optimization system will converge the spatial error of the model to be optimized to within the spatial error threshold in a cyclic manner, that is, the spatial error is less than the spatial error threshold. When the spatial error is still greater than the spatial error threshold, the model optimization system will use the model optimized in the current cycle as the model to be optimized, jump to the step of loading the model to be optimized into the target scene and the subsequent steps, re-optimize the model to be optimized, and execute the optimization cycle of the model to be optimized until the spatial error converges to within the spatial error threshold, completing the optimization of the model to be optimized and jumping out of the cycle.

[0055] In the embodiment of the present application, after the optimization of the model to be optimized is completed and the optimized model is obtained, the optimization results of the optimized model and the reference model are verified. When the verification result is that the spatial error is greater than the spatial error threshold, the optimized model is optimized again until the spatial error is less than the spatial error threshold, thereby improving the accuracy and effect of model optimization.

[0056] Since the system introduced in the second embodiment of the present application is a system used to implement the method of the first embodiment of the present application, based on the method introduced in the first embodiment of the present application, the person skilled in the art can understand the specific structure and deformation of the system, so it is not repeated here. All systems used in the method of the first embodiment of the present application belong to the scope of protection of this application.

[0057] Based on the same inventive concept, the present application also provides a third embodiment, referring to Figure 4 , Figure 4 This is a flowchart diagram of the third embodiment of the oblique photography LoD model optimization method based on differentiable rendering of the present application.

[0058] In this embodiment, the oblique photography LoD model optimization method based on differentiable rendering further includes steps S61 to S64: Step S61: determining a target level of a level below the highest accuracy level in the LoD model of the oblique photography; Step S62: Taking the target level as the starting point, traversing the LoD model in a manner of gradually reducing the model accuracy, and taking the current access level in the traversal process as the level to be optimized; Step S63: determining the reference level of the level to be optimized according to the reference level interval; Step S64: Acquire the reference model of the reference level and the model to be optimized of the level to be optimized.

[0059] It should be noted that in differentiable rendering, the higher the similarity between the reference model and the target model, the smaller the parameter change amplitude of the optimization algorithm gradient calculation, making the gradient change direction more accurate and less likely to fall into the local optimal solution. Among them, since the loss function value is relatively low, the amount of parameter change is smaller, and the speed of model fitting to a reasonable range is relatively high. Therefore, at the oblique photography LoD level, adjacent or similar LoD level models with higher similarity are used for fitting, thereby improving the convergence speed.

[0060] In this embodiment, the reference level interval is the number of levels between the target level to be optimized and the reference level of the higher level. The model optimization system optimizes the model of the lower level by taking the higher level model in the neighboring LoD level as the reference model.

[0061] For example, Figure 2 As shown, the image optimization system can determine the target level and / or reference level based on the level offset, and complete the optimization of all levels with the self-increment action of the level offset after the target level is optimized. After completing the optimization of the current target level, the image optimization system sets the target level to the next level by self-incrementing the level offset, for example, by adding one each time the optimization is completed.

[0062] Furthermore, based on the spatial region in the LoD model, the model optimization system further divides the model into different nodes and optimizes the model in units of nodes. Based on the data structure of the LoD model, models at different levels are set with corresponding mapping relationships. The model optimization system can determine the nodes to be optimized corresponding to the model in the target level based on the reference nodes in the reference model, or the nodes to be optimized in the target level can determine the corresponding reference nodes in the reference level.

[0063] In one embodiment, in order to solve the problem that the volume of high-detail level LoD data in the LoD level is too large and is not suitable for loading a large range of scenes, the model optimization system can perform model fitting from micro to macro. The model optimization system first selects the fitting area in a micro scene with a viewing angle close to the ground, loads a higher precision level as a reference scene, determines the reference node in the reference scene, loads a nearby lower precision level as a target scene in another rendering scene, and determines the node to be optimized corresponding to the reference node.

[0064] In another embodiment, the model optimization system can also select the corresponding node to be optimized from the unoptimized nodes of the model to be optimized according to the preset selection rules of the node to be optimized. Based on the tree data structure of the LoD model, in the reference level, the reference node corresponding to the child node of the node to be optimized is determined, and the optimized model of the node to be optimized and the reference model of the reference node are obtained.

[0065] For example, taking 3D Tiles oblique photography modeling as an example, in 3D Tiles, node data is organized in a quadtree, that is, each node corresponds to four child nodes, representing the four quadrants of space, including the upper left, upper right, lower left, and lower right. That is, one level node corresponds to four level nodes of the previous level. Therefore, to ensure the optimization effect of the model, before fitting the next level, it is necessary to ensure that the four nodes in the corresponding level are fitted. And so on, the fitting of adjacent levels from the highest accuracy to the lowest accuracy is completed progressively.

[0066] The present application optimizes the target level through the reference level of the adjacent level interval, thereby improving the optimization speed of the target level and avoiding the large level interval gap between the target level and the reference level, which leads to excessive data processing during the target level optimization process and causes excessive optimization time.

[0067] Since the system introduced in the third embodiment of the present application is a system used to implement the method of the first embodiment of the present application, based on the method introduced in the first embodiment of the present application, the person skilled in the art can understand the specific structure and deformation of the system, so it is not repeated here. All systems used in the method of the first embodiment of the present application belong to the scope of protection of the present application.

[0068] Based on the same inventive concept, the present application also provides a fourth embodiment, referring to Figure 5 , Figure 5 This is a flowchart diagram of the fourth embodiment of the oblique photography LoD model optimization method based on differentiable rendering of the present application.

[0069] In this embodiment, the oblique photography LoD model optimization method based on differentiable rendering further includes steps S65 to S67: Step S65: after completing the traversal of the LoD model, executing the self-increment action of the reference level interval; In this embodiment, in addition to fitting the model optimization through adjacent levels, the model optimization system can also fit similar levels in a progressive manner. After the model optimization system completes the optimization of each level of the LoD model based on the current reference level interval, it will perform a self-increment action of the reference level interval, that is, the reference level interval is increased by one.

[0070] For example, the initial value of the reference level interval is 0. After selecting the reference level and the target level, the number of levels between the target level and the reference level is 0, and they are adjacent levels. When the reference level interval is 2, it means that there are two levels between the reference level and the target level.

[0071] Step S66: when the reference level interval is less than the level interval threshold, jump to execute the step and subsequent steps of taking the target level as the starting point, traversing the LoD model in a manner of gradually reducing the model accuracy, and taking the current access level in the traversal process as the level to be optimized; Step S67: Alternatively, when the reference level interval is equal to or greater than the level interval threshold, the LoD model is hierarchically reorganized to obtain the model optimization result of the oblique photography.

[0072] In this embodiment, since the data results of oblique photography nodes are organized in a tree-like manner, the growth of the interval will cause the corresponding nodes of the reference level to grow geometrically, thereby causing the amount of data to grow geometrically. Therefore, the reference level interval is set with a reference level interval to avoid accessing too many level nodes during the optimization process of the model to be optimized.

[0073] For example, in 3D Tiles oblique photography modeling, model nodes are organized in a quadtree form, so the maximum value of the reference level interval can be selected as 3, that is, the level interval threshold is 4. When the reference level interval is 3, there are 256 reference nodes in the reference level to correspond to each node to be optimized. When the reference level interval is 4, there are 1024 reference nodes in the reference level to correspond to each node to be optimized. At this time, there are too many level nodes to be accessed by the reference node.

[0074] Specifically, Figure 2As shown in the figure, when the reference level interval is less than the level interval threshold, or the reference level interval is equal to or less than the maximum level interval, the model optimization system determines that there is an unoptimized reference level interval at this time, resets the level offset, re-traverses the detail level model based on the self-incremented reference level interval, and optimizes all levels. When the reference level interval is greater than or equal to the level interval threshold, or the reference level interval is greater than the maximum level interval, the model optimization system determines that there is no unoptimized reference level interval at this time, restores the format and reorganizes the level of the LoD model, and obtains the LoD model optimization result of oblique photography.

[0075] The embodiment of the present application gradually expands the interval between the target level and the reference level by self-incrementing the reference level interval, so that the target level is optimized based on the adjacent reference level to ensure the optimization speed, and the optimization effect of the model is improved in a step-by-step optimization manner. When the target level is optimized in a step-by-step manner, compared with directly optimizing each level with the highest level reference model, the optimization speed of the model can be significantly improved, while avoiding repeated optimization of the model details optimized in the highest level reference model.

[0076] Since the system introduced in the fourth embodiment of the present application is a system used to implement the method of the first embodiment of the present application, based on the method introduced in the first embodiment of the present application, the person skilled in the art can understand the specific structure and deformation of the system, so it is not repeated here. All systems used in the method of the first embodiment of the present application belong to the scope of protection of this application.

[0077] Based on the same inventive concept, the present application also provides a fifth embodiment, referring to Figure 6 , Figure 6 This is a flowchart diagram of the fifth embodiment of the oblique photography LoD model optimization method based on differentiable rendering of the present application.

[0078] In this embodiment, the oblique photography LoD model optimization method based on differentiable rendering further includes steps S01 to S04: Step S01: acquiring image data collected during the oblique photography process, identifying feature points in the image data, and determining positions of the feature points in the image data; Step S02: Based on the position of the feature point, the spatial position corresponding to the pixel point in the image data is calculated by triangulation to generate point cloud data; Step S03: converting the point cloud data into a triangular mesh model, and mapping the image data into the triangular mesh model to generate an oblique photography model; Step S04: constructing the LoD model of the corresponding level of the oblique photography model based on different viewing distances through a surface reduction algorithm, and associating the LoD model with the viewing distance.

[0079] In this embodiment, after the oblique photography LoD model obtains the image data through oblique photography, it can generate point cloud data based on the feature point positions corresponding to the feature points in the identified image and by triangulation. The spatial position corresponding to each pixel in the image is calculated. Based on the point cloud data, a corresponding triangular mesh model can be constructed through three-dimensional meshes and building materials data. The oblique photography model is generated by mapping the captured image data to the corresponding positions of the triangular mesh model. Furthermore, in order to divide the oblique photography model into models of different levels of detail through the LoD algorithm, the LoD algorithm constructs the LoD model of the corresponding level of the oblique photography model based on different viewing distances through a face reduction algorithm, and associates the LoD model with the viewing distance.

[0080] Optionally, in order to achieve the compatibility of the oblique photography modeling process and the LoD model generation process with the model optimization system, the model optimization system needs to convert the format of the triangular mesh data and material data in the model after obtaining the LoD model. For example, the triangular mesh in each node conversion of the grouped oblique photography LoD level is converted to obj format, and the corresponding material is converted to mtl. Correspondingly, after completing the optimization of all LoD models and before the reorganization of the LoD models, the model optimization system will also restore the format of the model data.

[0081] Since the system introduced in the fifth embodiment of the present application is a system used to implement the method of the first embodiment of the present application, based on the method introduced in the first embodiment of the present application, the person skilled in the art can understand the specific structure and deformation of the system, so it is not repeated here. All systems used in the method of the first embodiment of the present application belong to the scope of protection of the present application.

[0082] The present application provides an oblique photography LoD model optimization device based on differentiable rendering, the device comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the oblique photography LoD model optimization method based on differentiable rendering in the above-mentioned embodiment 1.

[0083] Reference below Figure 7, which shows a schematic diagram of the structure of the oblique photography LoD model optimization device based on differentiable rendering suitable for implementing the embodiment of the present application. The oblique photography LoD model optimization device based on differentiable rendering in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The oblique photography LoD model optimization device based on differentiable rendering is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0084] like Figure 7 As shown, the oblique photography LoD model optimization device based on differentiable rendering may include a processing device 1001 (such as a core processor, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In the random access memory 1004, various programs and data required for the operation of the oblique photography LoD model optimization device based on differentiable rendering are also stored. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the oblique photography LoD model optimization device based on differentiable rendering to communicate wirelessly or wired with other devices to exchange data. Although the oblique photography LoD model optimization device based on differentiable rendering with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0085] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0086] The oblique photography LoD model optimization device based on differentiable rendering provided by the present application adopts the oblique photography LoD model optimization method based on differentiable rendering in the above-mentioned embodiment, which can solve the technical problem of insufficient rendering accuracy of the oblique photography LoD model. Compared with the prior art, the beneficial effects of the oblique photography LoD model optimization device based on differentiable rendering provided by the present application are the same as the beneficial effects of the oblique photography LoD model optimization method based on differentiable rendering provided by the above-mentioned embodiment, and the other technical features in the oblique photography LoD model optimization device based on differentiable rendering are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0087] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0088] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0089] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the oblique photography LoD model optimization method based on differentiable rendering in the above-mentioned embodiment.

[0090] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, radio frequencies (RF, Radio Frequency), etc., or any suitable combination of the above.

[0091] The computer-readable storage medium may be included in the device for optimizing the oblique photography LoD model based on differentiable rendering; or may exist independently without being assembled into the device for optimizing the oblique photography LoD model based on differentiable rendering.

[0092] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the oblique photography LoD model optimization device based on differentiable rendering, the oblique photography LoD model optimization device based on differentiable rendering: obtains the model to be optimized and the reference model in the detail level LoD model of the oblique photography; loads the model to be optimized into the target scene, and loads the reference model into the reference scene; based on the photographic parameters of the oblique photography, renders the target scene and the reference scene through a virtual camera to obtain a target image and a reference image; calculates the spatial error between the target image and the reference image, and calculates the model optimization gradient corresponding to the spatial error through inverse rendering; optimizes the model to be optimized based on the model optimization gradient to obtain an optimized model.

[0093] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0094] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0095] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0096] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned oblique photography LoD model optimization method based on differentiable rendering, and can solve the technical problem of insufficient rendering accuracy of the oblique photography LoD model. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the oblique photography LoD model optimization method based on differentiable rendering provided in the above-mentioned embodiment, and will not be repeated here.

[0097] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for optimizing the LoD model of oblique photography based on differentiable rendering, characterized in that: The method comprises the following steps: In the oblique photography LoD model, obtain the model to be optimized and the reference model; Loading the model to be optimized into a target scene, and loading the reference model into a reference scene; Based on the photography parameters of the oblique photography, the target scene and the reference scene are rendered by a virtual camera to obtain a target image and a reference image; Calculating a spatial error between the target image and the reference image, and calculating a model optimization gradient corresponding to the spatial error through inverse rendering; Based on the model optimization gradient, the model to be optimized is optimized to obtain an optimized model.

2. The method according to claim 1, characterized in that After the step of optimizing the model to be optimized based on the model optimization gradient, the method further includes: Calculating a target space error between the optimization model and a reference model; When the target spatial error is equal to or greater than a spatial error threshold, taking the optimization model as the model to be optimized; Jump to the step of loading the model to be optimized into the target scene, and loading the reference model into the reference scene; Alternatively, when the target spatial error is less than the spatial error threshold, the optimization model is updated to the LoD model of the oblique photography.

3. The method according to claim 1, characterized in that The step of calculating the spatial error between the target image and the reference image, and calculating the model optimization gradient corresponding to the spatial error by inverse rendering comprises: Identify target pixels of the target image and map them to reference pixels in the reference image; Calculating a spatial error between the target image and the reference image according to the target pixel position and the reference pixel position; Based on the inverse rendering of the differentiable rendering, calculating the model optimization gradient of the spatial error relative to the target model parameters of the model to be optimized; The target model parameters are iteratively optimized by using a gradient descent algorithm and optimizing the gradient according to the model.

4. The method according to claim 1, characterized in that In the oblique photography LoD model, the steps of obtaining the model to be optimized and the reference model include: In the LoD model of the oblique photography, determining a target level of a level below the highest accuracy level; Taking the target level as the starting point, traversing the LoD model in a manner of gradually reducing the model accuracy, and taking the current access level in the traversal process as the level to be optimized; Determining the reference level of the level to be optimized according to the reference level interval; The reference model of the reference level and the model to be optimized of the level to be optimized are obtained.

5. The method according to claim 4, characterized in that After the step of taking the target level as the starting point, traversing the LoD model in a manner of gradually reducing the model accuracy, and taking the current access level in the traversal process as the level to be optimized, the method further includes: After completing the traversal of the LoD model, performing a self-increment action of the reference level interval; When the reference level interval is less than the level interval threshold, jump to execute the step and subsequent steps of taking the target level as the starting point, traversing the LoD model in a manner of gradually reducing the model accuracy, and taking the current access level in the traversal process as the level to be optimized; Alternatively, when the reference level interval is equal to or greater than the level interval threshold, the LoD model is hierarchically reorganized to obtain a model optimization result of the oblique photography.

6. The method according to claim 4, characterized in that The step of acquiring the reference model of the reference level and the model to be optimized of the level to be optimized comprises: According to a preset selection rule of the node to be optimized, selecting the corresponding node to be optimized from the unoptimized nodes of the model to be optimized; Based on the tree data structure of the LoD model, determining, in the reference level, a reference node corresponding to a child node of the node to be optimized; The model to be optimized of the node to be optimized and the reference model of the reference node are obtained.

7. The method according to claim 1, characterized in that The step of rendering the target scene and the reference scene by a virtual camera based on the photography parameters of the oblique photography to obtain the target image and the reference image comprises: Acquire the shooting angle of the oblique photography; Based on the shooting angles, constructing a target virtual camera array in the target scene and a reference virtual camera array in the reference scene; Rendering a front image and a side image of the target model through the target virtual camera array, and using the front image and the side image as the target image; And, rendering a front reference image and a side reference image of the reference model through the reference virtual camera array, and using the front reference image and the side reference image as the reference images.

8. The method according to claim 1, characterized in that In the oblique photography LoD model, before the step of obtaining the model to be optimized and the reference model, the method further includes: Acquire image data collected during the oblique photography process, identify feature points in the image data, and determine the positions of the feature points in the image data; Based on the position of the feature point, the spatial position corresponding to the pixel point in the image data is calculated by triangulation to generate point cloud data; Converting the point cloud data into a triangular mesh model, and mapping the image data into the triangular mesh model to generate an oblique photography model; By using a surface reduction algorithm, based on different viewing distances, the LoD model of the corresponding level of the oblique photography model is constructed, and the LoD model is associated with the viewing distance.

Images