Real orthophoto generation method based on neural network and surface confidence coefficient

Abstract

The invention provides a true orthophoto generation method and device based on a neural network and surface confidence, electronic equipment and a storage medium. According to the embodiment of the invention, the method comprises the steps: carrying out the confidence calculation of a high-quality three-dimensional model of a target geographic region, initializing a Gaussian elliptic disk, and obtaining an initial Gaussian elliptic disk set and a three-dimensional model surface confidence set of the target geographic region; performing confidence coefficient and surface continuity optimization on the initial Gaussian elliptic disk set through a neural network optimization model; rasterization is carried out on the optimized Gaussian elliptic disc set passing the test through an orthogonal projection method, and a true shoot image of the target geographic area is obtained. According to the method, a three-dimensional ellipsoid in a three-dimensional Gaussian splashing technology is abandoned, a two-dimensional elliptical disc is adopted as an element, the spatial distribution position and attitude of the Gaussian elliptical disc are controlled by using confidence and surface continuity constraint conditions, the surface continuity of the Gaussian model is greatly ensured, the quality of the Gaussian model is improved, and the application prospect is wide. And the accuracy of generating the true orthophoto is obviously improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for generating a true orthogonal image based on a neural network and surface confidence, an electronic device and a storage medium. Background Art

[0002] With the rapid development of drone and sensor technology, the cost of obtaining high-resolution and super-resolution data has become lower and lower, and it has gradually entered people's daily lives, subtly changing the way people obtain and process geographic information. In both military and civilian fields, true orthogonal imaging technology has shown great application prospects. In the military field, it can be used as an unmanned reconnaissance platform to provide battlefield situation awareness; in the civilian field, in addition to improving the efficiency of urban planning and environmental monitoring, it can also provide key information in disaster response. The update and iteration of true orthogonal imaging technology is the key to solving the problem of information acquisition in the above fields. Therefore, true orthogonal imaging technology has become the focus of attention in the field of geographic information system (GIS) and academia.

[0003] Currently, orthogonal rendering of 3D models is often used in the production of true orthogonal images. However, due to the limitations of the accuracy of 3D models, it is difficult to achieve photo-level accuracy in terms of details.

[0004] 3D Gaussian Splatting is a computer graphics technique for real-time radiation field rendering. This concept was first proposed in the 2023 paper "3D Gaussian Splatting for Real-Time Radiance Field Rendering". It achieves high-quality rendering effects by using 3D Gaussian to represent the scene and combining it with a fast visibility-aware rendering algorithm.

[0005] The Gaussian splatter model performs well in multi-view rendering and lighting simulation. However, if the Gaussian splatter model is directly used for orthogonal projection rasterization to obtain a true orthogonal image, the following problems exist:

[0006] Since the true orthogonal image is only obtained from a single perspective (usually vertically downward), the three-dimensional ellipsoid used in the traditional Gaussian splash model becomes redundant, which not only increases the computational complexity but also introduces unnecessary errors.

[0007] The projection of a 3D ellipsoid onto a 2D screen may not perfectly maintain the continuity of the original surface. In particular, when there are overlaps or intersections between ellipsoids, the boundary processing after projection may become complicated and difficult to control, resulting in discontinuities or jagged effects on the generated true projection image surface.

[0008] Since the Gaussian splatter model is constrained and trained only using images, its generalization ability is limited when rasterizing the model from a non-image perspective, resulting in poor rasterization results, blurring, distortion, or missing images. For example, when acquiring true orthogonal images of a certain geographic area, the images will appear incomplete and uneven. Summary of the invention

[0009] The embodiments of the present application provide a true orthogonal image generation method, electronic device and storage medium based on a neural network and surface confidence, so as to partially or completely solve the above-mentioned technical problems.

[0010] In a first aspect, an embodiment of the present application provides a method for generating a true orthogonal image based on a neural network and surface confidence, comprising:

[0011] Step S1, performing confidence calculation on a high-quality three-dimensional model of a target geographical area based on a confidence algorithm to obtain a three-dimensional model surface confidence set of the target geographical area, wherein the three-dimensional model surface confidence set of the target geographical area is constructed as a three-dimensional model surface confidence field of the target geographical area;

[0012] Step S2, initializing a Gaussian elliptical disk on the surface of the high-quality three-dimensional model of the target geographical area according to a preset true orthogonal image resolution, to obtain an initial Gaussian elliptical disk set of the target geographical area;

[0013] Step S3, constructing a first neural network optimization model with the confidence and surface continuity formula of the initial Gaussian elliptical disk as the loss error function, and optimizing the confidence and surface continuity of the initial Gaussian elliptical disk set based on the first neural network optimization model to obtain an optimized Gaussian elliptical disk set;

[0014] Step S4, rendering the optimized Gaussian elliptical disk set to obtain a rendered image, comparing the Gaussian model rendered image with the original image data of the target geographic area through image similarity, and when the similarity does not meet the preset first threshold, taking the optimized Gaussian elliptical disk as the initial Gaussian elliptical disk and re-optimizing from step S3; when the similarity meets the preset first threshold, taking the optimized Gaussian elliptical disk set as the final Gaussian elliptical disk set and entering step S5;

[0015] Step S5, according to the preset ground resolution of the target geographical area, the final Gaussian elliptical disk set is rasterized by orthogonal projection to obtain a true orthogonal image of the target geographical area, and the true orthogonal image is used as the regional image of the target geographical area for information management of the target geographical area.

[0016] In a second aspect, an embodiment of the present application provides a true orthogonal image generation device based on a neural network and surface confidence, comprising:

[0017] A surface confidence acquisition module, used to calculate the confidence of the high-quality three-dimensional model of the target geographical area based on a confidence algorithm to obtain a three-dimensional model surface confidence set of the target geographical area, wherein the three-dimensional model surface confidence set of the target geographical area is constructed as a three-dimensional model surface confidence field of the target geographical area;

[0018] An initial Gaussian elliptical disk set acquisition module is used to initialize a Gaussian elliptical disk on the surface of the high-quality three-dimensional model of the target geographical area according to a preset true orthogonal image resolution, so as to obtain an initial Gaussian elliptical disk set of the target geographical area;

[0019] A first neural network optimization module is used to construct a first neural network optimization model using a formula combining a maximum confidence constraint and a minimum surface continuity constraint of a Gaussian ellipse disk as a loss error function, and to perform confidence optimization and surface continuity optimization on an initial Gaussian ellipse disk set based on the first neural network optimization model to obtain an optimized Gaussian ellipse disk set;

[0020] An iterative module is used to render the optimized Gaussian elliptical disk set to obtain a rendered image, compare the Gaussian model rendered image with the original image data of the target geographical area through image similarity, and when the similarity does not meet the preset first threshold, use the optimized Gaussian elliptical disk as the initial Gaussian elliptical disk and re-optimize from the first neural network optimization module; when the similarity meets the preset first threshold, use the optimized Gaussian elliptical disk set as the final Gaussian elliptical disk set and enter the rasterization module;

[0021] The rasterization module is used to specify the ground resolution according to the preset target geographical area, rasterize the final Gaussian elliptical disk set by orthogonal projection method, and obtain a true orthogonal image of the target geographical area. The true orthogonal image is used as a regional image of the target geographical area for information management of the target geographical area.

[0022] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above methods when executing the computer program.

[0023] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above is implemented.

[0024] Compared with the prior art, this application has the following advantages:

[0025] According to an embodiment of the present application, the three-dimensional ellipsoid in the three-dimensional Gaussian splattering technology is abandoned and a two-dimensional elliptical disk is used as a primitive. A high-quality three-dimensional model is used to calculate the three-dimensional model surface confidence set of the target geographical area, and the Gaussian elliptical disk is initialized on the high-quality three-dimensional model surface of the target geographical area according to a preset true orthographic image resolution to obtain an initial Gaussian elliptical disk set of the target geographical area; a first neural network optimization model is constructed with the formula of the confidence and surface continuity of the initial Gaussian elliptical disk as the loss error function, and the confidence optimization and surface continuity optimization of the initial Gaussian elliptical disk set are performed based on the first neural network optimization model to obtain an optimized Gaussian elliptical disk set; after the optimization, the optimized Gaussian elliptical disk set is quality tested, if it does not meet the requirements, it continues to be optimized through the first neural network optimization model, if it meets the requirements, the optimized Gaussian elliptical disk set is used as the final Gaussian elliptical disk set, and the final Gaussian elliptical disk set is rasterized through the orthogonal projection method to obtain the true orthographic image of the target geographical area. By adopting the above scheme, by abandoning the three-dimensional ellipsoid in the three-dimensional Gaussian splashing technology and adopting the two-dimensional elliptical disk as the primitive, the problem that the three-dimensional ellipsoid may not perfectly maintain the continuity of the original surface when projected on the two-dimensional screen and the boundary processing after projection may become complicated and difficult to control is avoided. By using a high-quality three-dimensional model to calculate the three-dimensional surface confidence set and surface properties, the spatial distribution position and posture of the elliptical disk in the Gaussian model are controlled by using the formula of the minimum field strength confidence and the minimum surface continuity as constraints, the surface properties and continuity of the Gaussian model are greatly guaranteed, the quality of the Gaussian model is improved, and the accuracy of generating true orthophotos is significantly improved, which can not only meet the increasing demand for accurate acquisition of large-scale orthophotos brought about by the continuous development of image acquisition methods, but also meet the higher requirements of urban management departments, natural resource management departments or emergency management departments for data accuracy. In addition, the true orthophotos generated directly based on the results of three-dimensional modeling are improved, which are of low quality, poor visual effects, and prone to incompleteness and unevenness, and can meet the requirements of urban planning management, emergency response, resource survey and other projects for high-quality results. The method of using a two-dimensional elliptical disk as a primitive can reduce manual intervention, improve the quality of true radiographic results, ensure the uniformity of accuracy, and facilitate the matching of different results data in the later stage.

[0026] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0028] Figure 1 A schematic diagram of a scene of a method for generating true orthogonal images based on a neural network and surface confidence provided in this application;

[0029] Figure 2 A flowchart of a method for generating a true orthogonal image based on a neural network and surface confidence according to an embodiment of the present application;

[0030] Figure 3 is an example diagram of the surface confidence distribution of the three-dimensional model of the present application;

[0031] Figure 4 It is a comparison diagram of a true-projection image generated after optimization based on a neural network and a transmission true-projection image according to an embodiment of the present application;

[0032] Figure 5 is a structural block diagram of a true-reflective image generation device based on a neural network and surface confidence according to an embodiment of the present application; and

[0033] Figure 6 A block diagram of an electronic device used to implement an embodiment of the present application. DETAILED DESCRIPTION

[0034] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the concept or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary in nature and not restrictive.

[0035] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0036] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, without paying creative work, this specification can also be applied to other similar scenarios based on these drawings. Unless it is obvious from the voice environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0037] It should be understood that the "system", "device", "unit" and / or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0038] As shown in this specification and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0039] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0040] First, before specifically introducing the technical solution of the embodiment of the present disclosure, the technical background or technical evolution context on which the embodiment of the present disclosure is based is introduced. Generally, if a three-dimensional Gaussian splashing technique is used to generate a true orthogonal image, since the true orthogonal image is obtained from only a single perspective, the three-dimensional ellipsoid used in the three-dimensional Gaussian splashing technique becomes redundant, which increases the computational complexity and introduces unnecessary errors.

[0041] Based on the above background, the applicant provides a method for producing true photographic images based on a neural network through the embodiments of the present application to solve all or part of the above technical problems.

[0042] The following introduces the technical solutions involved in the embodiments disclosed in this application in combination with the scenarios to which the embodiments disclosed in this application are applied.

[0043] Figure 1 The figure is a schematic diagram of an exemplary application scenario for implementing the method of the embodiment of the present application.

[0044] like Figure 1As shown, through the preliminary data preparation and preprocessing, a high-quality three-dimensional model of the target geographical area is obtained; then, the high-quality three-dimensional model of the target geographical area is calculated and the Gaussian ellipse disk is initialized to obtain an initial Gaussian ellipse disk set and a surface confidence set of the three-dimensional model of the target geographical area; based on the image exterior orientation elements obtained simultaneously in the preliminary preparation, the initial Gaussian ellipse disk set is subjected to conventional image loss error optimization through the second neural network optimization model, and the optimized initial Gaussian ellipse disk set is subjected to confidence and surface continuity optimization through the first neural network optimization model to obtain an optimized Gaussian ellipse disk set; a quality test is performed on the optimized Gaussian ellipse disk, and when the quality test fails, the optimized Gaussian ellipse disk is used as the initial Gaussian ellipse disk to start optimization iteration from the second neural network optimization model; when the quality test passes, the optimized Gaussian ellipse disk set is used as the final Gaussian ellipse disk set, and the final Gaussian ellipse disk set is rasterized by the orthogonal projection method to obtain a true orthogonal image of the target geographical area.

[0045] The present application embodiment provides a method for generating a true orthogonal image based on a neural network and surface confidence. Figure 2 The flowchart of a method for generating a true orthogonal image based on a neural network and surface confidence according to an embodiment of the present application may include:

[0046] Step S1, calculate the confidence of the high-quality three-dimensional model of the target geographical area based on the confidence algorithm to obtain a three-dimensional model surface confidence set of the target geographical area, wherein the three-dimensional model surface confidence set of the target geographical area is constructed as a three-dimensional model surface confidence field of the target geographical area.

[0047] In order to improve the accuracy, reliability and practicality of the model, while enhancing the generalization ability and real-time performance of the model and providing support for decision-making, the method of calculating surface confidence is used to constrain the high-quality three-dimensional model and provide a quantitative indicator.

[0048] In order to better understand the concept of surface confidence, Figure 3 Shown is an example diagram of the surface confidence distribution of the three-dimensional model of the present application.

[0049] It is understandable that the surface confidence is used to constrain the position and posture of the Gaussian elliptical disk set. This is because, although there is a certain deviation between the three-dimensional model obtained by multi-view reconstruction technology and the surface of the real object, this deviation is relatively small, so the surface of the three-dimensional model can be considered to be basically credible. When optimizing the position and posture of the Gaussian elliptical disk set, if they deviate too much from the surface of the three-dimensional model (that is, they are in an area where the surface confidence is too low), it means that they may deviate too much from the surface of the real object. Therefore, in the optimization process of the neural network, the surface confidence is used to constrain the position and posture of the Gaussian elliptical disk set to ensure that they are in an area with a high surface confidence as much as possible.

[0050] In an optional embodiment, before step S1, original image data of the target geographical area is collected by aerial photography equipment; feature points of the original image data of the target geographical area are extracted by photogrammetry and aerial triangulation is performed on the original image data to obtain internal and external orientation elements of the image data of the target geographical area; multi-view reconstruction technology (Multi-View Stereo) is used to reconstruct a three-dimensional model of the image data that already has internal and external orientation elements to obtain a high-quality three-dimensional model of the target geographical area.

[0051] Aerial photography equipment can be collected by drones, which carry camera modules and information collection modules. Before the drone collects data, the flight path, take-off point, landing point, flight altitude, speed and other parameters are planned and set. After that, the drone pilot controls the drone in real time through the remote control, and the drone transmits data with the ground communication station. The collected raw image data includes raw image data, camera attitude information data, location information data, dense point cloud data, flight vectors, etc.

[0052] Step S2, according to a preset true orthogonal image resolution, initialize a Gaussian elliptical disk on the surface of the high-quality three-dimensional model of the target geographical area to obtain an initial Gaussian elliptical disk set of the target geographical area.

[0053] Since the purpose of this application is to generate a true orthogonal image, and the true orthogonal image is only obtained from a single perspective (usually vertically downward), the three-dimensional ellipsoid used in the traditional Gaussian splash model will become redundant, which not only increases the computational complexity, but also introduces unnecessary errors. In addition, the projection of the three-dimensional ellipsoid on the two-dimensional screen may not perfectly maintain the continuity of the original surface. In particular, when there is overlap or intersection between the ellipsoids, the boundary processing after the projection may become complicated and difficult to control, resulting in a discontinuous or jagged effect on the generated true orthogonal image surface. In order to reduce the computational complexity and improve the quality of the Gaussian model, in an optional embodiment, the initial Gaussian elliptical disk set is composed of multiple initial Gaussian elliptical disks; the initial Gaussian elliptical disk is the true orthogonal image resolution generated as needed as the sampling interval, and grid sampling is performed on the high-quality three-dimensional model surface of the target geographical area, and an initial Gaussian elliptical disk is generated at each sampling point coordinate; the long and short radii of the initial Gaussian elliptical disk are the sampling intervals, and the plane where the initial Gaussian elliptical disk is located is parallel to the tangent plane of the three-dimensional model surface at the point.

[0054] Step S3, constructing a first neural network optimization model with a formula combining the maximum confidence constraint and the minimum surface continuity constraint of the Gaussian ellipse disk as the loss error function, and performing confidence optimization and surface continuity optimization on the initial Gaussian ellipse disk set based on the first neural network optimization model to obtain an optimized Gaussian ellipse disk set.

[0055] In order to optimize the confidence and surface continuity of the Gaussian ellipse disk to improve the quality of the Gaussian model and the accuracy of the final generated true orthogonal image, in an optional implementation:

[0056] Step S301: construct a vertex set C based on the center coordinates of each Gaussian ellipse in the initial Gaussian ellipse set, where C = {C 1 , C 2 , C 3 , ..., C i}, where each vertex C i Represents a three-dimensional coordinate (X i , Y i , Z i ).

[0057] Step S302: Based on the high-quality three-dimensional model of the target geographic area, the nearest projection point set N of the vertex set C on the three-dimensional confidence field surface is calculated by a nearest distance algorithm, where N = {N 1 , N 2 , N 3 , ..., N i}.

[0058] In an optional embodiment, the method for calculating the nearest projection point includes establishing a k-nearest-neighbor tree structure for a vertex set C of the high-quality three-dimensional model of the target geographic area; calculating a nearest projection point based on the k-nearest-neighbor tree structure for each vertex C in the vertex set C. i Perform the nearest point query to obtain the distance vertex C i The nearest triangle Δ i ; Use the shortest distance algorithm from a three-dimensional point to a spatial triangle to obtain vertex C i To triangle Δ i The closest distance point on the i .

[0059] Step S303: the nearest projection point N in the nearest projection point set N i Normal of the triangle face i Normalize to get the surface normal vector NormN of the nearest projection point i , construct the evaluation normal set NormN;

[0060] NormN={NormN 1 ,NormN 2 ,NormN 3 , ..., NormN i};

[0061] Step S304, calculating a normal vector set NormC of the initial Gaussian ellipse disk set based on the major and minor axis vectors of the Gaussian ellipse disk, wherein the normal vector is represented as a vector perpendicular to the plane of the Gaussian ellipse disk;

[0062] NormC={NormC 1 ,NormC 2 ,NormC 3 , ..., NormC i}.

[0063] In an optional implementation, the normal vector set NormC is calculated as follows:

[0064] NormC i = normalization(A i ×B i )

[0065] Among them, A i is the major axis vector of the i-th Gaussian ellipse disk, B i is the minor axis vector of the i-th Gaussian ellipse disk, and normalization(,) means normalizing the vector.

[0066] Step S305, based on the Gaussian elliptical disk set, a surface continuity vector set is calculated by a surface continuity algorithm.

[0067] In an optional embodiment, the method for calculating the continuity set includes selecting k nearest neighbor points P in the neighborhood for each Gaussian elliptical disk in the initial Gaussian elliptical disk collection. j , calculate the mean μ of the neighborhood points, k is the specified parameter, the default is 10:

[0068]

[0069] Calculate the covariance matrix CM of the nearest points:

[0070]

[0071] Perform eigenvalue decomposition on the covariance matrix to obtain the matrix V:

[0072] CM=VΛV T

[0073] In the decomposed matrix V, the eigenvector VN with the smallest eigenvalue is selected as the surface continuity vector of the initial Gaussian elliptical disk.

[0074] This operation is performed on each Gaussian elliptical disk in the initial Gaussian elliptical disk set to obtain the surface continuity vector set VN corresponding to the Gaussian elliptical disk set = {VN 1 , VN 2 , VN 3 , ..., VN i}.

[0075] Step S306, calculating the confidence according to the position of each Gaussian elliptical disk in the initial Gaussian elliptical disk set in the surface confidence field, and taking the set of confidences of each initial Gaussian elliptical disk as the initial Gaussian elliptical disk confidence set E.

[0076] In an optional implementation, the Gaussian ellipse disk confidence is calculated using the following formula:

[0077] E i =exp(-k·|N i -C i |)

[0078] Where k is the attenuation coefficient and is a positive number, C i is the three-dimensional coordinate of the signal field disposed by the i-th Gaussian elliptical disk, N i is the coordinate of the nearest projection point of the i-th Gaussian ellipse disk on the field surface, and |...| represents the vector length of this vector.

[0079] Step S307, constructing a loss error function Loss of the first neural network optimization model based on the evaluation normal vector set, the normal vector set, the continuity set, and the confidence set F . The evaluation normal vector set is the normal vector set calculated from the field by the center point of the Gaussian ellipse disk and used for evaluation. The normal vector set refers to the normal vector set of the Gaussian ellipse disk.

[0080] In an optional implementation, the loss error function is calculated by the following formula:

[0081]

[0082] Where N is the total number of Gaussian elliptical disks; σ is the surface continuity weight factor; E i is the confidence of the i-th Gaussian ellipse disk, NormC i is the normal vector of the plane where the i-th Gaussian ellipse disk is located; NormN i is the evaluated normal vector of the i-th Gaussian ellipse disk; VN i is the surface continuity vector of the i-th Gaussian elliptical disk; |VN i | is the vector length of the surface continuity vector of the i-th Gaussian elliptical disk.

[0083] Step S308: based on the loss error function Loss F Construct the first neural network optimization model, iterate the first neural network model, and calculate the loss error function according to the loss error function. F The back propagation algorithm is used to calculate the gradient of the value and obtain the optimized Gaussian elliptical disk set.

[0084] Furthermore, in the generation of the three-dimensional model, that is, before step S3, in order to reduce the inaccurate three-dimensional Gaussian ellipse disk caused by image errors, in an optional implementation, a second neural network optimization model with image loss error as the loss function is constructed, and the parameters of the initial Gaussian ellipse disk set are optimized based on the second neural network optimization model. The initial Gaussian ellipse disk is rasterized using the camera posture in the corresponding collected original image data to obtain the scene rendering requirement map, for the aerial image; the parameters of the initial Gaussian ellipse disk set include center coordinates, ellipse major axis vector, ellipse minor axis vector, color and transparency.

[0085] The initial Gaussian ellipse disk is rasterized using the camera pose of the original image of the target geographic area to obtain the scene rendering requirement map. The original image data and the scene rendering requirement map are compared for similarity to obtain the image loss error. The back propagation algorithm is performed to calculate the gradient based on the calculated value of the following image loss error function, thereby optimizing the initial Gaussian ellipse disk:

[0086] Loss w-image=λ i ·Loss w +(1-λ i )·Loss w-SSIM

[0087] Among them, L w is the image loss error based on the L1 algorithm, L w-SSIM is the image loss function using weighted SSIM image similarity comparison, λ i is the image weight factor.

[0088] Step S4, rendering the optimized Gaussian elliptical disk set to obtain a rendered image, comparing the Gaussian model rendered image with the original image data of the target geographic area through image similarity, and when the similarity does not meet the preset first threshold, taking the optimized Gaussian elliptical disk as the initial Gaussian elliptical disk and re-optimizing from step S3; when the similarity meets the preset first threshold, taking the optimized Gaussian elliptical disk set as the final Gaussian elliptical disk set and entering step S5;

[0089] Step S5, according to the preset ground resolution of the target geographical area, the final Gaussian elliptical disk set is rasterized by orthogonal projection to obtain a true orthogonal image of the target geographical area, and the true orthogonal image is used as the regional image of the target geographical area for information management of the target geographical area.

[0090] In order to better show the true orthogonal image obtained by optimizing the true orthogonal image generation method based on neural network and surface confidence provided in this embodiment, Figure 4 The traditional true orthographic image is compared with the true orthographic image optimized by the neural network. Figure 4 In the figure, the left picture is the traditional true orthogonal image, and the right picture is the neural network optimized true orthogonal image. It can be seen that the transmission true orthogonal image is prone to incompleteness and streaking at the edge of the building, while the neural network optimized true orthogonal image will not have this problem at all, greatly improving the accuracy of the true orthogonal image.

[0091] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a true orthogonal image generation device based on a neural network and surface confidence. Figure 5 The figure is a structural block diagram of a true orthogonal image generation device based on a neural network and surface confidence according to an embodiment of the present application. The true orthogonal image generation device based on a neural network and surface confidence may include:

[0092] A surface confidence acquisition module 501 is used to calculate the confidence of the high-quality three-dimensional model of the target geographical area based on a confidence algorithm to obtain a three-dimensional model surface confidence set of the target geographical area, wherein the three-dimensional model surface confidence set of the target geographical area is constructed as a three-dimensional model surface confidence field of the target geographical area;

[0093] An initial Gaussian elliptical disk set acquisition module 502 is used to initialize a Gaussian elliptical disk on the high-quality three-dimensional model surface of the target geographic area according to a preset true orthogonal image resolution, so as to obtain an initial Gaussian elliptical disk set of the target geographic area;

[0094] The first neural network optimization module 503 is used to construct a first neural network optimization model with a formula combining the maximum confidence constraint and the minimum surface continuity constraint of the Gaussian ellipse disk as the loss error function, and to optimize the confidence and surface continuity of the initial Gaussian ellipse disk set based on the first neural network optimization model to obtain an optimized Gaussian ellipse disk set;

[0095] Iteration module 504 is used to render the optimized Gaussian elliptical disk set to obtain a rendered image, compare the Gaussian model rendered image with the original image data of the target geographical area through image similarity, and when the similarity does not meet the preset first threshold, use the optimized Gaussian elliptical disk as the initial Gaussian elliptical disk to re-optimize from the first neural network optimization module; when the similarity meets the preset first threshold, use the optimized Gaussian elliptical disk set as the final Gaussian elliptical disk set and enter the rasterization module;

[0096] The rasterization module 505 is used to specify the ground resolution according to the preset target geographical area, rasterize the final Gaussian elliptical disk set by orthogonal projection method, and obtain a true orthogonal image of the target geographical area. The true orthogonal image is used as a regional image of the target geographical area for information management of the target geographical area.

[0097] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0098] Figure 6 FIG. 1 is a block diagram of an electronic device used to implement an embodiment of the present application. Figure 6 As shown, the electronic device includes: a memory 610 and a processor 620. The memory 610 stores a computer program that can be run on the processor 620. When the processor 620 executes the computer program, the method in the above embodiment is implemented. The number of the memory 610 and the processor 620 can be one or more.

[0099] The electronic device also includes:

[0100] The communication interface 630 is used to communicate with external devices and perform data exchange transmission.

[0101] If the memory 610, the processor 620 and the communication interface 630 are implemented independently, the memory 610, the processor 620 and the communication interface 630 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0102] Optionally, in a specific implementation, if the memory 610, the processor 620 and the communication interface 630 are integrated on a chip, the memory 610, the processor 620 and the communication interface 630 can communicate with each other through an internal interface.

[0103] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0104] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0105] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0106] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0107] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting description, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0108] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0109] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0110] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0111] Any process or method described in the flow chart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0112] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or used in combination with these instruction execution systems, devices or apparatuses.

[0113] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0114] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0115] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope recorded in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for generating true orthogonal images based on neural networks and surface confidence, wherein: The method comprises: Step S1, performing confidence calculation on a high-quality three-dimensional model of a target geographical area based on a confidence algorithm to obtain a three-dimensional model surface confidence set of the target geographical area, wherein the three-dimensional model surface confidence set of the target geographical area is constructed as a three-dimensional model surface confidence field of the target geographical area; Step S2, initializing a Gaussian elliptical disk on the surface of the high-quality three-dimensional model of the target geographical area according to a preset true orthogonal image resolution, to obtain an initial Gaussian elliptical disk set of the target geographical area; Step S3, constructing a first neural network optimization model with a formula combining the maximum confidence constraint and the minimum surface continuity constraint of the Gaussian ellipse disk as the loss error function, and performing confidence optimization and surface continuity optimization on the initial Gaussian ellipse disk set based on the first neural network optimization model to obtain an optimized Gaussian ellipse disk set; Step S4, rendering the optimized Gaussian elliptical disk set to obtain a rendered image, comparing the Gaussian model rendered image with the original image data of the target geographic area through image similarity, and when the similarity does not meet the preset first threshold, taking the optimized Gaussian elliptical disk as the initial Gaussian elliptical disk and re-optimizing from step S3; when the similarity meets the preset first threshold, taking the optimized Gaussian elliptical disk set as the final Gaussian elliptical disk set and entering step S5; Step S5, according to the preset ground resolution of the target geographical area, the final Gaussian elliptical disk set is rasterized by orthogonal projection to obtain a true orthogonal image of the target geographical area, and the true orthogonal image is used as the regional image of the target geographical area for information management of the target geographical area.

2. The method according to claim 1, wherein: The initial Gaussian elliptical disk set is composed of multiple initial Gaussian elliptical disks; the initial Gaussian elliptical disk is a true orthogonal image resolution generated as needed as a sampling interval, grid sampling is performed on the high-quality three-dimensional model surface of the target geographic area, and an initial Gaussian elliptical disk is generated at each sampling point coordinate; the long and short radii of the initial Gaussian elliptical disk are the sampling interval, and the plane where the initial Gaussian elliptical disk is located is parallel to the tangent plane of the three-dimensional model surface at the point.

3. The method according to claim 1, wherein: The method further comprises: Collect original image data of the target geographical area through aerial photography equipment; Extracting feature points from the original image data of the target geographic area by photogrammetry and performing aerial triangulation processing on the original image data to obtain aerial triangulation image data of the target geographic area; A three-dimensional model is reconstructed based on the aerial triangulation image data of the target geographic area to obtain a high-quality three-dimensional model of the target geographic area.

4. The method according to claim 1, wherein: Before step S3, the following steps may also be included: A second neural network optimization model with image loss error as the loss function is constructed, and the parameters of the initial Gaussian elliptical disk set are optimized based on the second neural network optimization model; the parameters of the initial Gaussian elliptical disk set include center coordinates, ellipse major axis vector, ellipse minor axis vector, color and transparency.

5. The method according to claim 3, wherein: The step S3 comprises: Step S301: construct a vertex set C based on the center coordinates of each Gaussian ellipse in the initial Gaussian ellipse set, where C = {C1, C2, C3, ..., C i }, where each vertex C i Represents a three-dimensional coordinate (X i ,Y i ,Z i ); Step S302: Based on the high-quality three-dimensional model of the target geographic area, the nearest projection point set N of the vertex set C on the three-dimensional confidence field surface is calculated by a nearest distance algorithm, where N = {N1, N2, N3, ..., N i }; Step S303: the nearest projection point N in the nearest projection point set N i Normal of the triangle face i Normalize to get the surface normal vector NormN of the nearest projection point i , construct the evaluation normal set NormN; NormN={NormN1,NormN2,NormN3,...,NormN i }; Step S304, based on the major and minor axis vectors of the Gaussian ellipse disk, calculate the normal vector set NormC of the initial Gaussian ellipse disk set, where NormC = {NormC1, NormC2, NormC3, ..., NormC i ); Step S305, based on the initial Gaussian elliptical disk set, calculating the surface continuity vector set by using the surface continuity algorithm; Step S306, calculating the confidence according to the position of each Gaussian elliptical disk in the initial Gaussian elliptical disk set in the surface confidence field, and taking the set of confidences of each initial Gaussian elliptical disk as the initial Gaussian elliptical disk confidence set E; Step S307, constructing a loss error function Loss of the first neural network optimization model based on the evaluation normal vector set, the normal vector set, the continuity set, and the confidence set F ; Step S308: based on the loss error function Loss F Construct the first neural network optimization model, iterate the first neural network model, and calculate the loss error function according to the loss error function. F The back propagation algorithm is used to calculate the gradient of the value and obtain the optimized Gaussian elliptical disk set.

6. The method according to claim 5, wherein: The step S302 includes: Establishing a k-nearest-neighbor tree structure for a vertex set C of the high-quality three-dimensional model of the target geographic area; Based on the k-nearest neighbor tree structure, each vertex C in the vertex set C i Perform the nearest point query to obtain the distance vertex C i The nearest triangle Δ i ; Use the closest distance algorithm from the three-dimensional point to the spatial triangle to obtain vertex C i To triangle Δ i The closest distance point on the i .

7. The method according to claim 5, wherein: The normal vector set in step S304 is generated based on the following process: NormC i =normalization(A i ×B i ) Among them, A i is the major axis vector of the i-th Gaussian ellipse disk, B i is the minor axis vector of the i-th Gaussian ellipse disk, and normalization(,) means normalizing the vector.

8. The method according to claim 5, wherein: Step S305 includes: For each Gaussian elliptical disk in the initial Gaussian elliptical disk set, select the k nearest neighbor points P in the neighborhood j , calculate the mean μ of the neighborhood points: Calculate the covariance matrix CM of the nearest points: Perform eigenvalue decomposition on the covariance matrix to obtain the matrix V: CM=VΛV T In the decomposed matrix V, the eigenvector VN with the smallest eigenvalue is selected as the surface continuity vector of the initial Gaussian ellipse disk; This operation is performed on each Gaussian elliptical disk in the initial Gaussian elliptical disk set to obtain the surface continuity vector set VN corresponding to the Gaussian elliptical disk set = {VN1, VN2, VN3, ..., VN i }.

9. The method according to claim 5, wherein: The Gaussian ellipse disk confidence calculation process of step S306 include: E i =exp(-k·|N i -C i |) Where k is the attenuation coefficient and is a positive number, C i is the three-dimensional coordinate of the signal field disposed by the i-th Gaussian elliptical disk, N i are the coordinates of the nearest projection point of the i-th Gaussian elliptical disk on the field surface, and |...| represents the vector length of the calculation vector.

10. The method according to claim 5, wherein: The loss error function calculation process of step S307 includes: Where N is the total number of Gaussian elliptical disks; σ is the surface continuity weight factor; E i is the confidence of the i-th Gaussian ellipse disk, NormC i is the normal vector of the plane where the i-th Gaussian ellipse disk is located; NormN i is the evaluated normal vector of the i-th Gaussian ellipse disk; VN i is the surface continuity vector of the i-th Gaussian elliptical disk; |VN i | is the vector length of the surface continuity vector of the i-th Gaussian elliptical disk.

11. A true orthogonal image generation device based on neural network and surface confidence, wherein: The device comprises: A surface confidence acquisition module, used to calculate the confidence of the high-quality three-dimensional model of the target geographical area based on a confidence algorithm to obtain a three-dimensional model surface confidence set of the target geographical area, wherein the three-dimensional model surface confidence set of the target geographical area is constructed as a three-dimensional model surface confidence field of the target geographical area; An initial Gaussian elliptical disk set acquisition module is used to initialize a Gaussian elliptical disk on the surface of the high-quality three-dimensional model of the target geographical area according to a preset true orthogonal image resolution, so as to obtain an initial Gaussian elliptical disk set of the target geographical area; A first neural network optimization module is used to construct a first neural network optimization model using a formula combining a maximum confidence constraint and a minimum surface continuity constraint of a Gaussian ellipse disk as a loss error function, and to perform confidence optimization and surface continuity optimization on an initial Gaussian ellipse disk set based on the first neural network optimization model to obtain an optimized Gaussian ellipse disk set; An iterative module is used to render the optimized Gaussian elliptical disk set to obtain a rendered image, compare the Gaussian model rendered image with the original image data of the target geographical area through image similarity, and when the similarity does not meet the preset first threshold, use the optimized Gaussian elliptical disk as the initial Gaussian elliptical disk and re-optimize from the first neural network optimization module; when the similarity meets the preset first threshold, use the optimized Gaussian elliptical disk set as the final Gaussian elliptical disk set and enter the rasterization module; The rasterization module is used to specify the ground resolution according to the preset target geographical area, rasterize the final Gaussian elliptical disk set by orthogonal projection method, and obtain a true orthogonal image of the target geographical area. The true orthogonal image is used as a regional image of the target geographical area for information management of the target geographical area.

12. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 10 when executing the computer program.

13. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images