Method for generating true orthographic images based on neural networks and surface confidence

Abstract

The application provides a true orthophoto generation method and device based on a neural network and surface confidence, electronic equipment and a storage medium. According to the embodiment of the application, a high-quality three-dimensional model of a target geographic area is calculated by confidence and initialized as a Gaussian ellipse disc, an initial Gaussian ellipse disc set and a three-dimensional model surface confidence set of the target geographic area are obtained, the initial Gaussian ellipse disc set is optimized by a neural network optimization model, confidence and surface continuity are optimized, and the Gaussian ellipse disc set that passes the test after optimization is rasterized by an orthogonal projection method to obtain a true orthophoto of the target geographic area. The application discards a three-dimensional ellipsoid in a three-dimensional Gaussian splash technology and uses a two-dimensional ellipse disc as a base element, uses confidence and surface continuity constraints to control the spatial distribution position and posture of the Gaussian ellipse disc, greatly guarantees the surface continuity of the Gaussian model, improves the quality of the Gaussian model, and significantly improves the accuracy of generating a true orthophoto.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a real orthographic image generation method and device based on a neural network and surface confidence, an electronic device and a storage medium. BACKGROUND

[0002] With the rapid development of unmanned aerial vehicles and sensor technology, the cost of high-resolution, super-resolution data acquisition is becoming lower and lower, and it has gradually entered people's daily life, subtly changing people's way of obtaining and processing geographic information. In military and civilian fields, real orthographic image technology has shown great application prospects. In the military field, it can be used as an unmanned reconnaissance platform to provide battlefield situation awareness; in the civilian field, in addition to improving the efficiency of urban planning and environmental monitoring, it can also provide key information in disaster response. The update and iteration of real orthographic image technology is the key to solving the information acquisition problem in the above fields. Therefore, real orthographic image technology has become the focus of attention in the field of geographic information systems (GIS) and academia.

[0003] Currently, real orthographic image production often uses three-dimensional model orthogonal rendering methods, but is limited by the fact that three-dimensional model accuracy is difficult to achieve photo-level accuracy in details.

[0004] Three-dimensional Gaussian splatting technology (3D Gaussian Splatting) is a computer graphics technology used for real-time radiance field rendering. This concept was first proposed in the 2023 paper "3D Gaussian Splatting for Real-Time Radiance Field Rendering". It uses a 3D Gaussian representation of the scene, combined with a fast visibility-aware rendering algorithm, to achieve high-quality rendering results.

[0005] Gaussian splatting models perform well in multi-view rendering and lighting simulation, but if they are directly used for orthogonal projection rasterization to obtain real orthographic images, the following problems exist:

[0006] Since real orthographic images are only obtained from a single view (usually vertically downward), the three-dimensional ellipsoids used by traditional Gaussian splatting models become redundant, not only increasing computational complexity, but also introducing unnecessary errors.

[0007] The projection of three-dimensional ellipsoids on a two-dimensional screen may not perfectly maintain the continuity of the original surface. Especially when there is overlap or intersection between ellipsoids, the boundary processing after projection can become complex and difficult to control, resulting in discontinuous or jagged effects on the surface of the generated real orthographic image.

[0008] Due to the Gaussian splash model using only images for constraint and training, the generalization ability is limited when rasterizing the model from a non-image perspective, resulting in poor rasterization results, such as blurring, distortion, or missing, which can cause the image to be incomplete and not smooth when acquiring a true orthographic image of a certain geographic area. SUMMARY

[0009] Embodiments of the present application provide a true orthographic image generation method based on a neural network and surface confidence, an electronic device, and a storage medium to partially or completely solve the above technical problems.

[0010] In a first aspect, embodiments of the present application provide a true orthographic image generation method based on a neural network and surface confidence, comprising:

[0011] Step S1, performing confidence calculation on a high-quality three-dimensional model of a target geographic area based on a confidence algorithm to obtain a three-dimensional model surface confidence set of the target geographic area, wherein the three-dimensional model surface confidence set of the target geographic area is constructed as a three-dimensional model surface confidence field of the target geographic area;

[0012] Step S2, initializing a Gaussian ellipse disc on the surface of the high-quality three-dimensional model of the target geographic area according to a preset true orthographic image resolution to obtain an initial Gaussian ellipse disc set of the target geographic area;

[0013] Step S3, constructing a first neural network optimization model with a loss error function of the confidence and surface continuity formula of the initial Gaussian ellipse disc, and performing confidence optimization and surface continuity optimization on the initial Gaussian ellipse disc set based on the first neural network optimization model to obtain an optimized Gaussian ellipse disc set;

[0014] Step S4, rendering the optimized Gaussian ellipse disc set to obtain a rendered image, and comparing the Gaussian model rendered image with the original image data of the target geographic area through image similarity, when the similarity does not satisfy a preset first threshold, re-optimizing the optimized Gaussian ellipse disc as the initial Gaussian ellipse disc from step S3; when the similarity satisfies the preset first threshold, taking the optimized Gaussian ellipse disc set as the final Gaussian ellipse disc set and entering step S5;

[0015] Step S5, rasterizing the final Gaussian ellipse disc set through an orthogonal projection method according to a preset specified ground resolution of the target geographic area to obtain a true orthographic image of the target geographic area, and the true orthographic image is used as a regional image of the target geographic area for information management of the target geographic area.

[0016] In a second aspect, embodiments of the present application provide a true orthographic image generation device based on a neural network and surface confidence, comprising:

[0017] a surface confidence obtaining module configured to perform confidence calculation on the high-quality three-dimensional model of the target geographic region based on a confidence algorithm to obtain a three-dimensional model surface confidence set of the target geographic region, wherein the three-dimensional model surface confidence set of the target geographic region is constructed as a three-dimensional model surface confidence field of the target geographic region;

[0018] an initial Gaussian ellipse disc set obtaining module configured to initialize Gaussian ellipse discs on the surface of the high-quality three-dimensional model of the target geographic region according to a preset true orthographic resolution to obtain an initial Gaussian ellipse disc set of the target geographic region;

[0019] a first neural network optimization module configured to construct a first neural network optimization model with a formula combined with a maximum confidence constraint of the Gaussian ellipse disc and a minimum surface continuity constraint as a loss error function, perform confidence optimization and surface continuity optimization on the initial Gaussian ellipse disc set based on the first neural network optimization model, and obtain an optimized Gaussian ellipse disc set;

[0020] an iteration module configured to render the optimized Gaussian ellipse disc set to obtain a rendered image, compare the Gaussian model rendered image with original image data of the target geographic region through image similarity, re-optimize the optimized Gaussian ellipse disc as the initial Gaussian ellipse disc from the first neural network optimization module when the similarity does not satisfy a preset first threshold, and use the optimized Gaussian ellipse disc set as the final Gaussian ellipse disc set and input the final Gaussian ellipse disc set into a rasterization module when the similarity satisfies the preset first threshold;

[0021] a rasterization module configured to rasterize the final Gaussian ellipse disc set through an orthogonal projection method according to a preset specified ground resolution of the target geographic region to obtain a true orthographic image of the target geographic region, and use the true orthographic image as a regional image of the target geographic region for information management of the target geographic region.

[0022] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory. The processor implements any of the above methods when executing the computer program.

[0023] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement any of the above methods.

[0024] Compared with the prior art, the present application has the following advantages:

[0025] According to the embodiments of the present application, the three-dimensional ellipsoid in the three-dimensional Gaussian splash technology is abandoned and a two-dimensional elliptical disc is used as a primitive, a high-quality three-dimensional model is used to calculate a three-dimensional model surface confidence set of a target geographic area, and a Gaussian elliptical disc is initialized on the surface of the high-quality three-dimensional model of the target geographic area according to a preset true orthographic image resolution, to obtain an initial Gaussian elliptical disc set of the target geographic area; a first neural network optimization model is constructed with a formula of confidence and surface continuity of the initial Gaussian elliptical disc as a loss error function, and the initial Gaussian elliptical disc set is optimized in confidence and surface continuity based on the first neural network optimization model, to obtain an optimized Gaussian elliptical disc set; the optimized Gaussian elliptical disc set is tested for quality after optimization, if not meeting the requirements, the optimization is continued through the first neural network optimization model, if meeting the requirements, the optimized Gaussian elliptical disc set is taken as a final Gaussian elliptical disc set, and the final Gaussian elliptical disc set is rasterized through an orthogonal projection method, to obtain a true orthographic image of the target geographic area. By using the above scheme, the three-dimensional ellipsoid in the three-dimensional Gaussian splash technology is abandoned and a two-dimensional elliptical disc is used as a primitive, which avoids the problem that the three-dimensional ellipsoid may not perfectly maintain the continuity of the original surface when projected on a two-dimensional screen, and the boundary processing after projection may become complex and difficult to control. By using a high-quality three-dimensional model to calculate a three-dimensional surface confidence set and surface properties, and using a formula of minimum field strength confidence and minimum surface continuity as a constraint condition to control the spatial distribution position and attitude of the elliptical disc in the Gaussian model, the surface properties and continuity of the Gaussian model are greatly guaranteed, the quality of the Gaussian model is improved, the accuracy of generating the true orthographic image is significantly improved, the demand for accurate acquisition of large-scale orthographic images increasing with the continuous development of image acquisition means can be met, and higher requirements for data precision of city management departments, natural resource management departments or emergency management departments can be met. In addition, the true orthographic image generated directly based on three-dimensional modeling results is improved in quality and visual effect, and the problems of defects and non-smoothness are avoided, which can meet the requirements of city planning management, emergency response, resource investigation and other projects for high-quality results. The method of using a two-dimensional elliptical disc as a primitive can reduce manual intervention, improve the quality of true orthographic image results, guarantee the uniformity of precision, and facilitate the matching of different result data in the later stage.

[0026] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, the technical solutions can be implemented according to the content of the description, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described below. BRIEF DESCRIPTION OF DRAWINGS

[0027] In the drawings, like reference numerals refer to like elements throughout the various drawings. The drawings are not necessarily to scale, the emphasis instead being placed upon illustrating the principles of the application. It should be understood that the drawings are merely schematic and that the application can be embodied in many different forms.

[0028] Figure 1 A scene schematic diagram of a neural network and surface confidence based true orthographic image generation method provided by the present application;

[0029] Figure 2 A flowchart of a neural network and surface confidence based true orthographic image generation method of an embodiment of the present application;

[0030] Figure 3 A three-dimensional model surface confidence distribution example diagram of the present application;

[0031] Figure 4 A comparison diagram of a neural network and surface confidence based true orthographic image generated by an embodiment of the present application and a true orthographic image;

[0032] Figure 5 A structure block diagram of a neural network and surface confidence based true orthographic image generation device of an embodiment of the present application; and

[0033] Figure 6 A block diagram of an electronic device for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0034] In the following, only some exemplary embodiments are simply described. As those skilled in the art can recognize, the described embodiments can be modified in various different ways without departing from the concept or scope of the present application. Therefore, the drawings and the description are considered to be exemplary in nature, rather than limiting.

[0035] In order to facilitate understanding of the technical solutions of the embodiments of the present application, the related technologies of the embodiments of the present application are described below. The following related technologies can be combined with the technical solutions of the embodiments of the present application in any way as optional solutions, which all belong to the protection scope of the embodiments of the present application.

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiment description will be simply introduced below. Obviously, the drawings in the following description are only some examples or embodiments of the present application, and those skilled in the art can also apply the present application to other similar scenarios without creating any creative labor, unless it is obvious from the language environment or otherwise stated. The same reference numerals in the drawings represent the same structure or operation.

[0037] It should be understood that the terms "system," "device," "unit," and / or "module" as used herein are a method for distinguishing different components, elements, parts, sections, or assemblies from one another. However, the terms can be replaced by other expressions as long as the same meaning is conveyed.

[0038] As shown in the specification and claims herein, the words "a," "an," "one," and / or "the" are not meant to refer to only one thing, but can also mean one or more things unless the context clearly indicates otherwise. Generally, the terms "include," "including," and / or "comprising" are meant to be inclusive and not exclusive, and these terms do not constitute an exhaustive listing of steps or elements, and methods or devices can include other steps or elements.

[0039] Flowcharts are used in the specification to illustrate the operations performed by systems according to embodiments of the specification. It should be understood that the preceding or following operations are not necessarily performed in the order shown. Instead, various steps can be processed in reverse order or simultaneously. Other operations can also be added to or removed from these processes, or one or more steps can be removed from these processes.

[0040] First, before specifically introducing the technical solutions of the embodiments of the disclosure, the technical background or technical evolution context based on which the embodiments of the disclosure are introduced. Generally, if a true perspective image is generated by using a three-dimensional Gaussian splashing technology, since the true perspective image is obtained from a single view, the three-dimensional ellipsoid used by the three-dimensional Gaussian splashing technology will become redundant, which not only increases the calculation complexity, but also introduces unnecessary errors.

[0041] Based on the above background, the applicant provides a true photographic image production method based on a neural network through the embodiments of the present application to solve all or part of the above technical problems.

[0042] The technical solutions related to the embodiments of the present application will be introduced below in combination with the scene to which the embodiments of the present application are applied.

[0043] Figure 1 An exemplary application scene for implementing the method of the embodiments of the present application.

[0044] As Figure 1As shown, through the previous data preparation and preprocessing, a high-quality three-dimensional model of the target geographic area is obtained; then the high-quality three-dimensional model of the target geographic area is obtained through confidence calculation and initialization of the Gaussian ellipse disc, and an initial Gaussian ellipse disc set and a three-dimensional model surface confidence set of the target geographic area are obtained; based on the image exterior orientation elements obtained at the same time, the initial Gaussian ellipse disc set is optimized through a second neural network optimization model to perform regular image loss error optimization, and the optimized initial Gaussian ellipse disc set is optimized through a first neural network optimization model to perform confidence and surface continuity optimization, and an optimized Gaussian ellipse disc set is obtained; the optimized Gaussian ellipse disc is subjected to quality testing, when the quality testing fails, the optimized Gaussian ellipse disc is taken as the initial Gaussian ellipse disc to start optimization iteration from the second neural network optimization model; when the quality testing passes, the optimized Gaussian ellipse disc set is taken as the final Gaussian ellipse disc set, and the final Gaussian ellipse disc set is rasterized through the orthogonal projection method to obtain a true orthographic image of the target geographic area.

[0045] The embodiment of the application provides a true orthographic image generation method based on a neural network and surface confidence. Figure 2 As shown, the flowchart of the true orthographic image generation method based on the neural network and the surface confidence of the embodiment of the application can include:

[0046] In step S1, the confidence of the high-quality three-dimensional model of the target geographic area is calculated based on the confidence algorithm, and a three-dimensional model surface confidence set of the target geographic area is obtained, wherein the three-dimensional model surface confidence set of the target geographic area is constructed as a three-dimensional model surface confidence field of the target geographic area.

[0047] In order to improve the accuracy, reliability and practicability of the model, and at the same time enhance the generalization ability and real-time performance of the model, the method of calculating the surface confidence is used to constrain the high-quality three-dimensional model, and a quantitative index is provided.

[0048] In order to better understand the concept of surface confidence, as shown in Figure 3 As shown is an example diagram of the three-dimensional model surface confidence of the application.

[0049] It can be understood that the surface confidence is used to constrain the position and pose of the set of Gaussian ellipsoids. This is because, although there is a certain deviation between the three-dimensional model obtained by the multi-view reconstruction technology and the surface of the real object, the deviation is relatively small, and therefore the surface of the three-dimensional model can be considered to be basically reliable. When optimizing the position and pose of the set of Gaussian ellipsoids, if they deviate too much from the surface of the three-dimensional model (i.e., in a region with too low surface confidence), it means that they may deviate too much from the surface of the real object. Therefore, in the optimization process of the neural network, the surface confidence is used to constrain the position and pose of the set of Gaussian ellipsoids, so as to ensure that they are in a region with high surface confidence as much as possible.

[0050] In an optional embodiment, before step S1, raw image data of a target geographic area is collected by a aerial photography device; feature point extraction is performed on the raw image data of the target geographic area by photogrammetry, and an aerial triangulation process is performed on the raw image data to obtain internal and external orientation elements of the image data of the target geographic area; a three-dimensional model is reconstructed on the image data having the internal and external orientation elements using a multi-view reconstruction technology (Multi-View Stereo) to obtain a high-quality three-dimensional model of the target geographic area.

[0051] The aerial photography device can use a drone for collection, wherein the drone carries a camera module and an information collection module. Before the drone collects, flight path, takeoff point, landing point, flight height, speed and other parameters are planned and set. Then the pilot of the drone controls the drone in real time through a remote controller, and the drone transmits data with a ground communication station. The collected raw image data includes raw image data, camera pose information data, position information data, dense point cloud data, flight vector, etc.

[0052] Step S2, according to a predetermined true orthographic image resolution, initializing a Gaussian ellipsoid on the surface of the high-quality three-dimensional model of the target geographic area to obtain an initial set of Gaussian ellipsoids of the target geographic area.

[0053] Since the purpose of this application is to generate true orthogonal images, which are obtained from only a single viewpoint (usually vertically downward), the 3D ellipsoids used in traditional Gaussian splash models become redundant, increasing computational complexity and introducing unnecessary errors. Furthermore, the projection of the 3D ellipsoid onto a 2D screen may not perfectly maintain the continuity of the original surface. Especially when there is overlap or intersection between ellipsoids, the boundary processing after projection can become complex and difficult to control, resulting in discontinuous or jagged effects on the generated true orthogonal image surface. To reduce computational complexity and improve the quality of the Gaussian model, in one optional implementation, the initial Gaussian elliptical disk set consists of multiple initial Gaussian elliptical disks; the initial Gaussian elliptical disks are used as the sampling interval based on the required resolution of the generated true orthogonal image, and mesh sampling is performed on the high-quality 3D model surface of the target geographic area, generating an initial Gaussian elliptical disk at each sampling point coordinate; the major and minor radii of the initial Gaussian elliptical disks are the sampling interval, and the plane containing the initial Gaussian elliptical disk is parallel to the tangent plane of the 3D model surface at that point.

[0054] Step S3: Construct a first neural network optimization model with the formula combining the maximum confidence constraint and the minimum surface continuity constraint of the Gaussian elliptical disk as the loss error function. Based on the first neural network optimization model, perform confidence optimization and surface continuity optimization on the initial set of Gaussian elliptical disks to obtain an optimized set of Gaussian elliptical disks.

[0055] To optimize the confidence level and surface continuity of the Gaussian elliptical disk, thereby improving the quality of the Gaussian model and the accuracy of the final generated true orographic image, in one optional implementation:

[0056] Step S301: Based on the center coordinates of each Gaussian elliptical disk in the initial set of Gaussian elliptical disks, construct a vertex set C, C = {C1, C2, C3, ..., C6}. i}, where each vertex C i Represents a three-dimensional coordinate (X) i Y i Z i ).

[0057] Step S302: Based on the high-quality 3D model of the target geographic region, calculate the set N of the nearest projection points of vertex set C on the 3D confidence field surface using the nearest distance algorithm, where N = {N1, N2, N3, ..., N...} i}

[0058] In one optional implementation, the calculation of the nearest projection point includes establishing a k-nearest neighbor tree structure for the vertex set C of the high-quality 3D model of the target geographic region; and calculating the nearest projection point for each vertex C in the vertex set C based on the k-nearest neighbor tree structure. iPerform a nearest point query to obtain the distance from vertex C. i The nearest triangle Δ i Using the closest distance algorithm from a 3D point to a spatial triangle, vertex C is obtained. i To triangle Δ i The nearest point on the surface, i.e., the nearest projection point N. i .

[0059] Step S303, for the nearest projection point N in the set of nearest projection points N i Norm of the triangle i Normalize the vector to obtain the surface normal vector NormN of the nearest projection point. i Construct the evaluation normal set NormN;

[0060] NormN={NormN1,NormN2,NormN3,...,NormN i};

[0061] Step S304: Calculate the set of normal vectors NormC of the initial Gaussian elliptical disk set based on the major and minor axis vectors of the Gaussian elliptical disk, where the normal vector is represented as a vector perpendicular to the plane of the Gaussian elliptical disk.

[0062] NormC={NormC1,NormC2,NormC3,...,NormC i}

[0063] In one alternative implementation, the set of normal vectors NormC is calculated as follows:

[0064] NormC i =normalization(A i ×B i )

[0065] Among them, A i Let B be the major axis vector of the i-th Gaussian elliptic disk. i Let be the minor axis vector of the i-th Gaussian elliptic disk, and normalization() represents the normalization process for the vector.

[0066] Step S305: Based on the Gaussian elliptical disk set, calculate the surface continuity vector set using the surface continuity algorithm.

[0067] In one alternative implementation, the method for calculating the continuity set includes selecting k nearest neighbors P within the neighborhood of each Gaussian elliptical disk in the initial set of Gaussian elliptical disks. j Calculate the mean μ of the neighborhood points, where k is a specified parameter, with a default value of 10:

[0068]

[0069] Compute the covariance matrix CM of the closest points:

[0070]

[0071] Eigenvalue decomposition of the covariance matrix, obtain the matrix V:

[0072] CM = VAV T

[0073] Select the smallest eigenvalue VN in the matrix V obtained by decomposition as the surface continuity vector of the initial Gaussian ellipse disc.

[0074] Perform this operation on each Gaussian ellipse disc in the initial Gaussian ellipse disc set, and obtain the surface continuity vector set VN = {VN1, VN2, VN3,..., VN i} corresponding to the Gaussian ellipse disc set.

[0075] Step S306, calculate the confidence according to the position of each Gaussian ellipse disc in the surface confidence field, and take the confidence set of each initial Gaussian ellipse disc as the initial Gaussian ellipse disc confidence set E.

[0076] In an optional embodiment, the Gaussian ellipse disc confidence is calculated by the following formula:

[0077] E i = exp(-k·|N i -C i |)

[0078] Where k is the attenuation coefficient and is positive, C i is the three-dimensional coordinate of the i-th Gaussian ellipse disc in the confidence field, N i is the coordinate of the nearest projection point of the i-th Gaussian ellipse disc on the field surface, and |...| represents the vector length of the vector.

[0079] Step S307, based on the evaluation normal vector set, the normal vector set, the continuity set, and the confidence set, construct the loss error function Loss F of the first neural network optimization model. The evaluation normal vector set is calculated from the field by the Gaussian ellipse disc center point, and is used for evaluation. The normal vector set refers to the normal vector set of the Gaussian ellipse disc.

[0080] In an optional embodiment, the loss error function is calculated by the following formula:

[0081]

[0082] wherein N is the total number of Gaussian ellipses; σ is a surface continuity weight factor; E i is the confidence of the i-th Gaussian ellipse; NormC i is the normal vector of the plane where the i-th Gaussian ellipse lies; NormN i is the evaluation normal vector of the i-th Gaussian ellipse; VN i is the surface continuity vector of the i-th Gaussian ellipse; |VN i is the vector length of the surface continuity vector of the i-th Gaussian ellipse.

[0083] Step S308, based on the loss error function Loss F , a first neural network optimization model is constructed, and the first neural network model is iterated, and the gradient is calculated by executing the back propagation algorithm according to the value of the loss error function Loss F , and an optimized Gaussian ellipse set is obtained.

[0084] Further, in the three-dimensional model generation, that is, before step S3, in order to reduce the inaccurate three-dimensional Gaussian ellipses caused by image errors, in an optional embodiment, a second neural network optimization model is constructed with an image loss error as a loss function, and the parameters of the initial Gaussian ellipse set are optimized based on the second neural network optimization model, and the initial Gaussian ellipses are rasterized using the camera poses in the corresponding collected original image data to obtain a scene rendering requirement graph for the aerial image; the parameters of the initial Gaussian ellipse set include the center coordinates, the ellipse long axis vector, the ellipse short axis vector, the color and the transparency.

[0085] The initial Gaussian ellipses are rasterized using the camera poses of the target geographic area original image to obtain a scene rendering requirement graph, the similarity of the original image data and the scene rendering requirement graph is compared, the image loss error is obtained, and the gradient is calculated by executing the back propagation algorithm according to the value calculated by the following image loss error function, so as to optimize the initial Gaussian ellipses:

[0086] Loss w-image = λ i · Loss w + (1 - λ i ) · Loss w-SSIM

[0087] wherein L w is the image loss error based on the L1 algorithm, L w-SSIM is the image loss function using the weighted SSIM image similarity comparison, λ i is an image weight factor.

[0088] Step S4: Render the optimized Gaussian elliptical disk set to obtain a rendered image. Compare the rendered image of the Gaussian model with the original image data of the target geographic area using image similarity. If the similarity does not meet a preset first threshold, use the optimized Gaussian elliptical disk as the initial Gaussian elliptical disk and re-optimize from step S3. If the similarity meets the preset first threshold, use the optimized Gaussian elliptical disk set as the final Gaussian elliptical disk set and proceed to step S5.

[0089] Step S5: Based on the preset target geographic region and the specified ground resolution, the final Gaussian elliptical disk set is rasterized using orthogonal projection to obtain the true orthogonal image of the target geographic region. The true orthogonal image is used as the regional image of the target geographic region for information management of the target geographic region.

[0090] To better demonstrate the optimized true orographic image obtained by the true orographic image generation method based on neural networks and surface confidence provided in this embodiment, the following steps are taken: Figure 4 A comparison was made between traditional true orthophotos and true orthophotos optimized by neural networks. Figure 4 In the image, the left image is a traditional true orthophoto, and the right image is a neural network-optimized true orthophoto. It can be seen that the traditional true orthophoto is prone to defects and streaks at the building edges, while the neural network-optimized true orthophoto completely avoids these problems, greatly improving the accuracy of the true orthophoto.

[0091] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a true orthophoto generation apparatus based on neural networks and surface confidence. For example... Figure 5 The diagram shown is a structural block diagram of a true orthophoto generation apparatus based on neural networks and surface confidence according to an embodiment of this application. This true orthophoto generation apparatus based on neural networks and surface confidence may include:

[0092] The surface confidence acquisition module 501 is used to calculate the confidence of a high-quality 3D model of a target geographic region based on a confidence algorithm, and obtain a set of surface confidence of the 3D model of the target geographic region, wherein the set of surface confidence of the 3D model of the target geographic region is constructed as a surface confidence field of the 3D model of the target geographic region.

[0093] The initial Gaussian elliptic disk set acquisition module 502 is used to initialize Gaussian elliptic disks on the surface of a high-quality 3D model of the target geographic region according to a preset true orthophoto resolution, and obtain the initial Gaussian elliptic disk set of the target geographic region.

[0094] The first neural network optimization module 503 is configured to construct a first neural network optimization model in which a formula combined with a maximum confidence constraint of a Gaussian ellipse disc and a minimum surface continuity constraint is taken as a loss error function, perform confidence optimization and surface continuity optimization on an initial Gaussian ellipse disc set based on the first neural network optimization model, and obtain an optimized Gaussian ellipse disc set.

[0095] The iteration module 504 is configured to render the optimized Gaussian ellipse disc set to obtain a rendered image, compare a Gaussian model rendered image with original image data of a target geographic region by image similarity, re-optimize the optimized Gaussian ellipse disc as an initial Gaussian ellipse disc from the first neural network optimization module when the similarity does not satisfy a preset first threshold, and take the optimized Gaussian ellipse disc set as a final Gaussian ellipse disc set and input the final Gaussian ellipse disc set into the rasterization module when the similarity satisfies the preset first threshold.

[0096] The rasterization module 505 is configured to rasterize the final Gaussian ellipse disc set by an orthogonal projection method according to a preset specified ground resolution of the target geographic region, and obtain a true image of the target geographic region, which is taken as a region image of the target geographic region for information management of the target geographic region.

[0097] The functions of each module in each device in the embodiments of the present application can be referred to the corresponding description in the above method, and have the corresponding beneficial effects, which will not be repeated here.

[0098] Figure 6 A block diagram of an electronic device for implementing the embodiments of the present application is shown in FIG. 6. As shown in FIG. 6, the electronic device includes a memory 610 and a processor 620, and the memory 610 stores a computer program that can run on the processor 620. The processor 620 implements the method in the above embodiments when executing the computer program. The number of the memory 610 and the processor 620 can be one or more. Figure 6

[0099] The electronic device further includes:

[0100] A communication interface 630 for communicating with external devices and performing data transmission.

[0101] ​If the memory 610, the processor 620 and the communication interface 630 are implemented independently, the memory 610, the processor 620 and the communication interface 630 can be connected to each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 6 Only one thick line is used to represent the bus in the figure, but it does not mean that there is only one bus or only one type of bus.

[0102] Optionally, in a specific implementation, if the memory 610, the processor 620 and the communication interface 630 are integrated on a chip, the memory 610, the processor 620 and the communication interface 630 can complete communication between each other through an internal interface.

[0103] The embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method provided in the embodiment of the present application.

[0104] The embodiment of the present application further provides a chip, which includes a processor, is used for calling and running instructions stored in a memory, and makes a communication device installed with the chip execute the method provided in the embodiment of the present application.

[0105] The embodiment of the present application further provides a chip, which includes an input interface, an output interface, a processor and a memory, the input interface, the output interface, the processor and the memory are connected through an internal connection path, and the processor is used for executing code in the memory, when the code is executed, the processor is used for executing the method provided in the embodiment of the present application.

[0106] It is to be understood that the above-mentioned processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. It is to be noted that the processor can be a processor supporting an advanced RISC machine (ARM) architecture.

[0107] Further, the memory can include a read-only memory and a random access memory, optionally. The memory can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), or a flash memory, for example. The volatile memory can include a random access memory (RAM) used as an external cache. Many forms of RAM are available by way of example but not limitation. For example, a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous DRAM (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a SyncLink DRAM (SLDRAM), and a direct Rambus RAM (DR RAM) can be used.

[0108] In the above-described embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded on a computer, all or part of the processes or functions according to the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium.

[0109] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, a person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0110] In addition, the terms "first", "second", etc. are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified.

[0111] Any process or method described in the flowchart or otherwise described herein can be understood as a representation of code including one or more executable instructions for performing a specific logical function or process. Also, the scope of the preferred embodiments of the present application includes additional implementations that can not be shown or discussed explicitly, including implementations in which functions are performed in different orders, in substantially simultaneous fashion, or in reverse order according to the functions involved.

[0112] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as a list of executable instructions for implementing the logic function, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus, such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from the instruction execution system, device or apparatus, or in conjunction with these instructions execution system, device or apparatus.

[0113] It should be understood that each part of the present application can be realized by hardware, software, firmware or a combination thereof. In the above embodiments, a plurality of steps or methods can be realized by software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above-mentioned embodiment methods can be completed by a program instructing the relevant hardware, which can be stored in a computer readable storage medium and includes one or a combination of the steps of the embodiment methods when executed.

[0114] In addition, each functional unit in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of a software functional module. The above-mentioned integrated module, if realized in the form of a software functional module and sold or used as an independent product, can also be stored in a computer readable storage medium. The storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0115] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for generating a true orthographic image based on a neural network and surface confidence, wherein, The method comprises: Step S1, performing confidence calculation on a high-quality three-dimensional model of a target geographic area based on a confidence algorithm to obtain a three-dimensional model surface confidence set of the target geographic area, wherein the three-dimensional model surface confidence set of the target geographic area is constructed as a three-dimensional model surface confidence field of the target geographic area; Step S2, initializing a Gaussian ellipse disc on the surface of the high-quality three-dimensional model of the target geographic area according to a preset true orthographic image resolution to obtain an initial Gaussian ellipse disc set of the target geographic area; Step S3, obtaining an evaluation normal set, a normal vector set, a surface continuity vector set, a confidence set based on the initial Gaussian ellipse disc set, and constructing a loss error function Loss of the first neural network optimization model F ; constructing a first neural network optimization model based on the loss error function Loss F ; performing a back propagation algorithm to calculate a gradient according to a value of the loss error function Loss F , and obtaining an optimized Gaussian ellipse disc set; wherein a confidence is calculated according to a position of each Gaussian ellipse disc in the surface confidence field, and a confidence set of each initial Gaussian ellipse disc is taken as an initial Gaussian ellipse disc confidence set E; the loss error function Loss F of the first neural network optimization model specifically includes: N is the total number of Gaussian ellipse discs; σ is a surface continuity weight factor; E i is a confidence of the i-th Gaussian ellipse disc; NormC i is a normal vector of a plane where the i-th Gaussian ellipse disc is located; NormN i is an evaluation normal vector of the i-th Gaussian ellipse disc; VN i is a surface continuity vector of the i-th Gaussian ellipse disc; |VN i | is a vector length of the surface continuity vector of the i-th Gaussian ellipse disc; Step S4, rendering the optimized Gaussian ellipse disc set to obtain a rendered image, comparing the Gaussian model rendered image with original image data of the target geographic area through image similarity, when the similarity does not satisfy a preset first threshold, reoptimizing the optimized Gaussian ellipse disc as an initial Gaussian ellipse disc from step S3; when the similarity satisfies the preset first threshold, taking the optimized Gaussian ellipse disc set as a final Gaussian ellipse disc set and entering step S5; Step S5, rasterizing the final Gaussian ellipse disc set through an orthogonal projection method according to a preset specified ground resolution of the target geographic area to obtain a true orthographic image of the target geographic area, which is used as a regional image of the target geographic area for information management of the target geographic area.

2. The method of claim 1, wherein, The initial Gaussian ellipse disc set is composed of a plurality of initial Gaussian ellipse discs; the initial Gaussian ellipse disc is generated according to a required true orthographic image resolution as a sampling interval, grid sampling is performed on the surface of the high-quality three-dimensional model of the target geographic area, and an initial Gaussian ellipse disc is generated at each sampling point coordinate; the major and minor radii of the initial Gaussian ellipse disc are the sampling interval, and the plane where the initial Gaussian ellipse disc is located is parallel to the tangent plane of the three-dimensional model surface at the point.

3. The method of claim 1, wherein, The method further comprises: collecting original image data of the target geographic area through aerial photography equipment; extracting feature points from the original image data of the target geographic area through photogrammetry and performing aerial triangulation processing on the original image data to obtain aerial triangulation image data of the target geographic area; performing three-dimensional model reconstruction on the aerial triangulation image data of the target geographic area to obtain a high-quality three-dimensional model of the target geographic area.

4. The method of claim 1, wherein, Before step S3, the method can further comprise: constructing a second neural network optimization model with an image loss error as a loss function, and optimizing parameters of the initial Gaussian ellipse disc set based on the second neural network optimization model; the parameters of the initial Gaussian ellipse disc set include center coordinates, ellipse major axis vectors, ellipse minor axis vectors, color and transparency.

5. The method of claim 3, wherein, Step S3 comprises: Step S301, based on the center coordinates of each Gaussian ellipse disc in the initial Gaussian ellipse disc set, constructing a vertex set C, C = {C1, C2, C3, …, Cn}, wherein each vertex C i represents a three-dimensional coordinate (X i ,Y i ,Z i ). i ​ Step S302, according to the high-quality three-dimensional model of the target geographic area, the nearest distance algorithm is used to calculate the nearest projection point set N of the vertex set C on the three-dimensional confidence field surface, N={N1, N2, N3, …, Nn}, n is the number of points in the vertex set C. i}; Step S303, normalizing the normal of the triangle face where the nearest projection point N in the nearest projection point set N i i Step S304, obtaining the surface normal vector NormN of the nearest projection point by normalizing the normal Norm of the triangle face where the nearest projection point N i Step S305, constructing the evaluation normal set NormN​ NormN={NormN1,NormN2,NormN3,…,NormN i }; Step S304, calculating the normal vector set NormC of the initial Gaussian ellipse disc set based on the long and short axis vectors of the Gaussian ellipse disc, NormC={NormC1, NormC2, NormC3, …, NormC i}; Step S305, calculating a surface continuity vector set based on the initial Gaussian ellipse disc set through a surface continuity algorithm.

6. The method of claim 5, wherein, Step S302 comprises: establishing a k-nearest neighbor tree structure for a vertex set C of the high-quality three-dimensional model of the target geographic area; For each vertex C in the vertex set C, based on the k-neighbor tree structure, the vertex C is determined as a query point, and a nearest point query is performed to obtain a vertex C closest to the vertex C i The vertex C closest to the vertex C is determined as a vertex of a triangle Δ closest to the vertex C i The triangle Δ closest to the vertex C is determined as a triangle closest to the vertex C i ; Using a nearest distance algorithm of a three-dimensional point to a spatial triangle, the vertex C is obtained i to the nearest distance point on the triangle Δ i , i.e. the nearest projection point N i .

7. The method of claim 5, wherein, The normal vector set of step S304 is generated based on the following process: NormC i = normalization(A i × B i ) where A i is the long axis vector of the i-th Gaussian ellipse disk, B i is the short axis vector of the i-th Gaussian ellipse disk, and normalization(,) denotes normalization of a vector.

8. The method of claim 5, wherein, Step S305 comprises: Selecting k nearest neighbors P in the neighborhood for each Gaussian Ellipse Disc in the initial set of Gaussian Ellipse Discs j Calculating the mean value μ of the neighborhood points calculating a covariance matrix CM of the nearest points: Eigenvalue decomposition is performed on the covariance matrix to obtain a matrix V: CM = VΛV T In the decomposed matrix V, select the smallest eigenvector VN in the eigenvalue as the surface continuity vector of the initial Gaussian ellipse disc; This operation is performed on each of the initial set of Gaussian ellipses to obtain a set of surface continuity vectors VN corresponding to the set of Gaussian ellipses VN = {VN1, VN2, VN3, …, VN i}.

9. The method of claim 1, wherein, The Gaussian ellipse disc confidence calculation process of step S3 includes: The device includes: E i = exp(-k · |N i -C i |) where k is an attenuation coefficient and is positive, C i is the three-dimensional coordinate of the i-th Gaussian elliptical disc in the confidence field, N i is the coordinate of the nearest projection point of the i-th Gaussian elliptical disc on the field surface, |...| denotes the vector length of the calculated vector.

10. A neural network and surface confidence based ground truth image generation apparatus, wherein, A surface confidence acquisition module is configured to perform confidence calculation on a high-quality three-dimensional model of a target geographic region based on a confidence algorithm to obtain a three-dimensional model surface confidence set of the target geographic region, wherein the three-dimensional model surface confidence set of the target geographic region is constructed as a three-dimensional model surface confidence field of the target geographic region. An initial Gaussian ellipse disc set acquisition module is configured to initialize a Gaussian ellipse disc on the high-quality three-dimensional model surface of the target geographic region according to a preset true orthophoto resolution to obtain an initial Gaussian ellipse disc set of the target geographic region. An iteration module is configured to render the optimized Gaussian ellipse disc set to obtain a rendered image, compare the Gaussian model rendered image with original image data of the target geographic region through image similarity, and when the similarity does not satisfy a preset first threshold, re-optimize the optimized Gaussian ellipse disc as an initial Gaussian ellipse disc from the first neural network optimization module; when the similarity satisfies the preset first threshold, take the optimized Gaussian ellipse disc set as a final Gaussian ellipse disc set and enter a rasterization module. The first neural network optimization module is configured to obtain an evaluation normal set, a normal vector set, a surface continuity vector set, a confidence set based on the initial Gaussian ellipse disc set, and construct a loss error function Loss of the first neural network optimization model F ; based on the loss error function Loss F , the first neural network optimization model is constructed, the first neural network model is iterated, and a gradient is calculated by performing a back propagation algorithm according to a value of the loss error function Loss F , and an optimized Gaussian ellipse disc set is obtained; wherein the confidence is calculated according to a position of each Gaussian ellipse disc in a surface confidence field, and a confidence set of each initial Gaussian ellipse disc is taken as an initial Gaussian ellipse disc confidence set E; the loss error function Loss of the first neural network optimization model F Specifically includes: N is the total number of Gaussian ellipse discs; σ is a surface continuity weight factor; E i is the confidence of the i-th Gaussian ellipse disc; NormC i is a normal vector of a plane in which the i-th Gaussian ellipse disc is located; NormN i is an evaluation normal vector of the i-th Gaussian ellipse disc; VN i is a surface continuity vector of the i-th Gaussian ellipse disc; |VN i | is a vector length of the surface continuity vector of the i-th Gaussian ellipse disc; A rasterization module is configured to rasterize the final Gaussian ellipse disc set through an orthogonal projection method according to a preset specified ground resolution of the target geographic region to obtain a true orthophoto of the target geographic region, and the true orthophoto is used as a regional image of the target geographic region for information management of the target geographic region. 11.An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-9. 12.A computer readable storage medium having a computer program stored therein, wherein the computer program, when executed by a processor, implements the method of any one of claims 1-9. ​

Images