Three-dimensional reconstruction method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention discloses a three-dimensional reconstruction method. The method comprises the following steps: acquiring a satellite image containing a target area; obtaining a target area image and a camera model parameter corresponding to the target area image by using the satellite image; constructing a point cloud by using the target area image and camera model parameters corresponding to the target area image; performing three-dimensional Gaussian splashing processing on the point cloud to obtain a three-dimensional real scene model of the target area; wherein the original point of a camera coordinate system used by the three-dimensional Gaussian splash processing is determined according to camera model parameters. The satellite image is used as an image input source for three-dimensional reconstruction, a whole three-dimensional reconstruction process suitable for the satellite image is constructed based on the three-dimensional Gaussian splashing technology, and three-dimensional reconstruction with the satellite image as the input source is achieved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a three-dimensional reconstruction method and device. Background Art

[0002] With the rapid development of real-life 3D modeling and artificial intelligence technology, the process of creating 3D models has been greatly simplified, and the quality of 3D rendering has been significantly improved. High-precision 3D rendering can now show the real world we live in with amazing realism, depict the precise details of social infrastructure, reproduce the exquisite design of historical buildings, and even capture the magnificent scenery of the natural environment. These revolutionary changes have not only promoted the rapid development of multiple fields such as virtual reality, augmented reality and game development, but have also been widely used in industries such as architecture, urban planning and geographic information systems (GIS).

[0003] However, three-dimensional reconstruction of large scenes usually requires huge costs. Traditional technologies based on oblique photogrammetry require the use of drones to perform aerial photography of scenes that require three-dimensional models. When the modeling scope reaches the city level, this method is obviously time-consuming and labor-intensive, and is not conducive to mass replication. Summary of the invention

[0004] The present application provides a three-dimensional reconstruction method and device to facilitate quick and easy three-dimensional reconstruction of large scenes.

[0005] This application provides the following solutions:

[0006] According to a first aspect, a three-dimensional reconstruction method is provided, the method comprising: acquiring a satellite image containing a target area; obtaining a target area image and camera model parameters corresponding to the target area image using the satellite image; constructing a point cloud using the target area image and the camera model parameters corresponding to the target area image; performing three-dimensional Gaussian splash processing on the point cloud to obtain a three-dimensional real-scene model of the target area; wherein the origin of the camera coordinate system used for the three-dimensional Gaussian splash processing is determined based on the camera model parameters.

[0007] According to a second aspect, a three-dimensional reconstruction device is provided, the device comprising: a satellite image acquisition unit, which acquires a satellite image containing a target area; a target area acquisition unit, which uses the satellite image to obtain a target area image and camera model parameters corresponding to the target area image; a point cloud construction unit, which uses the target area image and the camera model parameters corresponding to the target area image to construct a point cloud; and a Gaussian reconstruction unit, which performs three-dimensional Gaussian splash processing on the point cloud to obtain a three-dimensional real scene model of the target area; wherein the origin of the camera coordinate system used for the three-dimensional Gaussian splash processing is determined based on the camera model parameters.

[0008] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0009] According to a fourth aspect, an electronic device is provided, comprising: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, and the program instructions, when read and executed by the one or more processors, execute the steps of any one of the methods described in the first aspect.

[0010] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0011] The application adopts satellite images as the image input source for 3D reconstruction, and makes adaptive adjustments to the satellite images based on the 3D Gaussian splashing technology. The origin of the camera coordinate system used for the 3D Gaussian splashing processing is determined according to the camera model parameters, so that the origin of the camera coordinate system in the 3D Gaussian splashing algorithm can be set according to the camera model parameters of the satellite image, thereby realizing the synthesis of a 3D real scene model of the target area under the satellite orthographic perspective, greatly reducing the cost of 3D reconstruction, and is not restricted by time and space, and can quickly and easily realize 3D reconstruction for large scenes.

[0012] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 A system architecture diagram applicable to the embodiments of the present application;

[0015] Figure 2 A flowchart of a three-dimensional reconstruction method provided in an embodiment of the present application;

[0016] Figure 3 A schematic diagram of the process of the three-dimensional reconstruction method provided in an embodiment of the present application;

[0017] Figure 4 A structural block diagram of a three-dimensional reconstruction device provided in an embodiment of the present application;

[0018] Figure 5 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0020] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0021] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0022] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0023] There are already some 3D reconstruction technologies that can generate 3D models with rich details by using tilted cameras to shoot ground targets from different angles. However, this technology strongly relies on manual refinement and manual labeling to complete modeling, which limits the efficiency of 3D reconstruction and has high labor costs.

[0024] In view of this, the present application provides a new idea. In order to facilitate the understanding of the present application, the system architecture on which the present application is based is first described. Figure 1 An exemplary system architecture to which the embodiments of the present application can be applied is shown. Figure 1 As shown in , the system architecture may include a user device and a three-dimensional reconstruction device located at a map application server.

[0025] The user can input a three-dimensional reconstruction request through the user device, and the user device sends the request to the three-dimensional reconstruction device of the map application server. The map application server requests and obtains satellite images from satellite image servers, databases, etc. The three-dimensional reconstruction device can use the method provided in the embodiment of the present application to perform three-dimensional reconstruction on the target area in the satellite image to obtain a three-dimensional real scene model. The server side of the map application can respond to the request of the user terminal and send the three-dimensional real scene model of the target area to the user terminal, and the user terminal uses the three-dimensional real scene model for rendering to obtain a two-dimensional image, a three-dimensional image, a VR (virtual reality) scene or an AR (augmented reality) scene, etc.

[0026] Among them, user equipment may include but is not limited to: smart mobile terminals, smart home devices, wearable devices, PC (Personal Computer), etc. Among them, smart mobile devices may include such as mobile phones, tablet computers, laptops, PDAs (Personal Digital Assistants), Internet cars, etc. Smart home devices may include smart TVs, smart refrigerators, etc. Wearable devices may include such as smart watches, smart glasses, virtual reality devices, augmented reality devices, mixed reality devices (i.e., devices that can support virtual reality and augmented reality), etc.

[0027] The 3D reconstruction device can be set up as an independent server, a server group, or a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product in the cloud computing service system to solve the problems of difficult management and weak service scalability in traditional physical hosts and virtual private servers (VPS). Figure 1 In addition to the illustrated architecture, the 3D reconstruction device may also be disposed on a computer terminal with relatively strong computing capabilities.

[0028] As one of the possible implementation methods, a user can input a 3D reconstruction request through a user device, and the user device sends the request to a 3D reconstruction device on the server side through a network. After the 3D reconstruction device generates a 3D real scene model for the target area in the satellite image, the 3D real scene model is returned to the user device through the network, and the user device displays it to the user.

[0029] It should be understood that Figure 1 The user equipment and the 3D reconstruction device in the figure are only for illustration. Any number of user equipment and 3D reconstruction devices may be provided according to implementation requirements.

[0030] Figure 2 A flow chart of a three-dimensional reconstruction method provided in an embodiment of the present application. The method can be performed by Figure 1 The three-dimensional reconstruction device in the system shown is performed. Figure 2 As shown in , the method may include the following steps:

[0031] Step 201: Acquire a satellite image containing a target area.

[0032] Step 202: Obtain a target area image and camera model parameters corresponding to the target area image using satellite images.

[0033] Step 203: construct a point cloud using the target area image and the camera model parameters corresponding to the target area image.

[0034] Step 204 , performing three-dimensional Gaussian splash processing on the point cloud to obtain a three-dimensional real scene model of the target area; wherein the origin of the camera coordinate system used in the three-dimensional Gaussian splash processing is determined according to the camera model parameters.

[0035] The application adopts satellite images as the image input source for 3D reconstruction, and makes adaptive adjustments to the satellite images based on the 3D Gaussian splashing technology. The origin of the camera coordinate system used for the 3D Gaussian splashing processing is determined according to the camera model parameters, so that the origin of the camera coordinate system in the 3D Gaussian splashing algorithm can be set according to the camera model parameters of the satellite image, thereby realizing the synthesis of a 3D real scene model of the target area under the satellite orthographic perspective, greatly reducing the cost of 3D reconstruction, and is not restricted by time and space, and can quickly and easily realize 3D reconstruction for large scenes.

[0036] First, the above step 201, namely "obtaining a satellite image including the target area", is described in detail in conjunction with the embodiment.

[0037] This application aims to restore the three-dimensional real scene model of the target area through satellite images. Satellite images can be obtained in many ways, for example, through public data platforms such as the Digital Earth Open Platform and the Geospatial Data Cloud, which provide a large number of remote sensing data satellite images. They can also be acquired by different satellite platforms or image acquisition devices such as sensors at different times. The satellite images acquired in this way have the characteristics of multi-source and multi-temporal.

[0038] The target area is the area required for three-dimensional reconstruction in this application, and the target area is a partial area contained in the satellite image. This application obtains multiple satellite images, one of which may contain the entire area of ​​the target area, or may only contain a partial area of ​​the target area. In addition to the geographic environment information of the target area, the satellite image may also contain a variety of band information, such as visible light bands, infrared bands, etc. These band information can be used to assist the subsequent reconstruction process and provide necessary prior information. In order to ensure the quality of the final reconstruction, the satellite image can also include different imaging angles and provide multi-viewing information. Among them, different imaging angles can be achieved by adjusting the swing of the satellite. While acquiring the satellite image, the geometric correction parameters of the image can also be obtained, such as the RPC (Rational Polynomial Coefficient) parameters of the remote sensing camera model, which are used to convert pixel coordinates into geographic coordinates.

[0039] The above step 202, namely "obtaining the target area image and the camera model parameters corresponding to the target area image by using the satellite image", is described in detail below in conjunction with the embodiments.

[0040] The target area exists in a part of the satellite image. By processing the satellite image, an image of the target area can be obtained. The above processing can be to segment the satellite image according to the range of the target area to obtain the image of the target area. Artificial intelligence technologies such as deep learning can also be used for image extraction to effectively identify and extract the target area from the satellite image.

[0041] As a preferred embodiment, before using satellite images to obtain images of the target area, the satellite images can be preprocessed to improve the quality of the acquired images of the target area. The preprocessing operations may include: performing radiation correction on satellite images to eliminate the effects of atmospheric and sensor inconsistencies on satellite images; performing defogging on satellite images to remove the effects of atmospheric scattering on satellite images; performing geometric correction on satellite images to correct positional deviations in satellite images caused by factors such as the curvature of the earth and terrain undulations; performing panchromatic fusion on satellite images to combine high-resolution panchromatic images with low-resolution multispectral images to reduce spectral distortion and enhance spatial information; performing super-resolution processing on satellite images to make the images appear clearer and more detailed through super-resolution technology, thereby improving the quality of the final reconstruction. The satellite images are preprocessed using at least one of the above preprocessing operations to obtain a preprocessed image, and then the target area image is obtained from the preprocessed image.

[0042] The camera model parameters of the target area image are obtained from the target area image. The feature points in the image can be first extracted and matched, the camera's intrinsic parameter matrix is ​​solved by a mathematical optimization method, and the camera's extrinsic parameter matrix is ​​solved by combining the world coordinates of the feature points with the image coordinates. The camera parameters can also be automatically obtained by using existing calibration tools or computer vision libraries (such as OpenCV, MATLAB's Computer Vision Toolbox, etc.). The camera model parameters can also be obtained using a pre-trained machine learning model.

[0043] As a preferred implementation, the RPC parameters corresponding to the target area image may be obtained, and the camera model parameters of the target area image may be obtained based on the RPC parameters corresponding to the target area image.

[0044] RPC parameters are a set of parameters used to describe the geometric and optical characteristics of the camera during the remote sensing image acquisition process. They are used to describe the method of the geometric imaging model of satellite images and establish the geometric relationship between satellite images and geographic coordinates. It is a key component of remote sensing image processing. It includes multiple parameters such as satellite attitude and ground control points. These parameters can be used for geometric correction and geographic coding of images.

[0045] RPC parameters are usually provided by the satellite imagery vendor, and they may be included in the imagery's metadata file or provided as a separate file. These parameters can be extracted from the imagery's raw data products or obtained through the vendor's data processing services. RPC parameters can also be read and applied using geospatial data processing libraries such as GDAL (Geospatial Data Abstraction Library, GDAL), which provides RPC-based coordinate conversion functions that can convert row and column numbers to longitude and latitude, and vice versa.

[0046] The camera model defines the relationship between the 3D scene and the 2D image, including the position, orientation, and internal parameters of the camera. The camera model parameters of the target area image can be obtained according to the RPC parameters. The extracted RPC parameters can be converted into a usable format for camera modeling, including converting the parameters from radians to degrees and applying necessary conversions, such as conversion from the XYZ coordinate system to ECEF (Earth-Centered, Earth-Fixed, Earth-centered rectangular coordinate system); the converted RPC parameters can be used to create a camera model, and libraries such as OpenCV, GLM, or ESRI's Camera Model NHG617 can be used.

[0047] In particular, the camera model parameters of the present application can add tilt parameters on the basis of the traditional pinhole camera model parameters. When the camera model parameters are subsequently used to construct the point cloud, the tilt parameters can better provide multi-view information of the satellite image and improve the quality of three-dimensional reconstruction. Satellite images, satellite orbits and attitude data can provide information about the sensor tilt. By analyzing these data, the tilt parameters in the RPC model can be estimated, thereby obtaining the tilt parameters in the camera model.

[0048] Figure 3 A schematic diagram of the process of the three-dimensional reconstruction method provided in the embodiment of the present application is shown in FIG. Figure 3 As shown, when the present application is segmented, the satellite image or preprocessed image can be segmented within the scope of the target area to obtain multiple slices (tiles), and the multiple slices are used as the target area image. Among them, the size of each slice can be determined according to a predefined size, and each slice can be appropriately expanded on the basis of the predefined size, so that the content of the slice is partially overlapped, so that when the slice is reconstructed in three dimensions, the visible faults at the slice seams can be reduced or eliminated, thereby improving the visual effect. Afterwards, according to the RPC parameters corresponding to the multiple slices, the camera model parameters corresponding to the multiple slices are obtained, and the camera model parameters corresponding to the multiple slices are the camera model parameters of the target area image.

[0049] The above step 203, namely "constructing a point cloud using the target area image and the camera model parameters corresponding to the target area image", is described in detail below in conjunction with an embodiment.

[0050] The present application constructs a point cloud using a target area image and camera model parameters of the target area image. When the target area image is a plurality of slices, the present application constructs a corresponding point cloud for each slice image.

[0051] Point clouds can be constructed in a variety of ways, for example, using image data from multiple perspectives to generate point clouds through a stereo matching algorithm; or using a deep learning model to generate a three-dimensional point cloud from a two-dimensional image.

[0052] As an implementable method, feature matching can be performed on the target area images corresponding to multiple perspectives; the camera model parameters of the target area images and the result of feature matching are used to perform motion structure recovery SFM (Structure from Motion, SFM) to obtain a sparse point cloud. Multi-view stereo MVS (Multi-View Stereo, MVS) estimation is performed using the sparse point cloud to generate a dense point cloud with a density greater than the above sparse point cloud.

[0053] SFM is a computer vision technology used to recover the 3D structure of a scene and the motion state of the camera from a set of static images without prior information. It infers the motion trajectory of the camera and the 3D structure of the scene by analyzing the similarity between images. This application can use the 3D reconstruction software colmap to perform SFM on the target area image, generate optimized camera pose parameters and sparse point cloud of the target area, and provide a basis for subsequent 3D modeling.

[0054] MVS is also a computer vision technology used to calculate the three-dimensional structure of a scene from multiple images taken from different perspectives. MVS mainly estimates the depth information of the same object or scene element by comparing it from different perspectives, thereby generating point cloud data or a three-dimensional mesh model. The core idea of ​​this technology is to use the parallax between multiple images to infer the depth information of the object, and then construct a three-dimensional model of the scene. When the present application uses sparse point clouds for multi-perspective stereo MVS estimation and generates dense point clouds, specifically: for target area images corresponding to multiple perspectives, use sparse point clouds for MVS estimation to obtain the depth of each pixel in the target area images corresponding to multiple perspectives, and generate depth maps corresponding to multiple perspectives; based on the relationship between the depth confidence corresponding to each point in the depth map and the confidence threshold, the depth information of the depth map is screened; the depth maps corresponding to the screened multiple perspectives are fused to obtain a dense point cloud.

[0055] Among them, tools such as colmap or GeoMVSNet can be used to perform MVS estimation on sparse point clouds to generate depth maps corresponding to multiple perspectives. The depth map contains the depth information of each pixel, where the depth information includes the depth value and the depth confidence. The depth confidence is the information that characterizes the credibility of the depth value.

[0056] The present application can filter depth information of depth maps corresponding to multiple perspectives according to the filtering parameters, and the filtering parameters are determined according to the depth information estimation status corresponding to each depth map. After obtaining the depth information corresponding to the depth map, after determining the filtering parameters according to the depth information, other depth maps are projected onto the current depth map, and then depth information filtering is performed. After traversing all depth maps, the depth information filtering result is obtained.

[0057] As an implementable method, the screening parameters may include a confidence threshold, and screening the depth information of the depth maps corresponding to multiple perspectives based on the confidence threshold includes: determining the confidence threshold of each depth map based on the depth confidence distribution corresponding to each depth map; filtering out the depth information in each depth map whose confidence is less than or equal to the corresponding confidence threshold.

[0058] When selecting some or all of the points to form a dense point cloud based on the relationship between the depth confidence and the confidence threshold corresponding to each point in the fused depth map, depth information with a depth confidence greater than or equal to the confidence threshold can be screened out from the depth map. This can also be understood as deleting the depth information with a depth confidence less than the confidence threshold in the depth map.

[0059] Among them, the confidence threshold can be a preset fixed value, or it can be determined based on the depth confidence distribution corresponding to each depth map. Specifically, the statistical data (such as the mean, median or standard deviation) of the depth confidence distribution can be calculated, and these statistical data can be used to determine the confidence threshold. Preferably, the confidence threshold of the depth map can be determined based on the depth confidence corresponding to the probability peak in the probability distribution of the depth confidence of the depth map. In the probability distribution of the depth confidence, the confidence corresponding to the peak is usually the depth confidence corresponding to the largest number of pixels in the overall distribution, and the depth confidence can be used as the confidence threshold of the depth map.

[0060] As another practicable manner, the screening parameter may include a distance error threshold, and the distance error threshold of each depth map is determined based on the shooting distance of the target area image to which each depth map belongs.

[0061] For the depth information of pixel point i on the depth map corresponding to viewing angle a, if the distance between the depth information of pixel point i obtained by mapping the depth maps corresponding to other viewing angles to viewing angle a exceeds the distance error threshold of the depth map corresponding to viewing angle a, indicating that the depth information error of pixel point i is relatively large, then the depth information of pixel point i on the depth map corresponding to viewing angle a is filtered out, viewing angle a is any viewing angle among multiple viewing angles, and pixel point i is any point on the target area image corresponding to viewing angle a.

[0062] The distance error threshold of the depth map is positively correlated with the shooting distance of the target area image to which the depth map belongs. The positive correlation can be linear or nonlinear. For example, when the shooting distance is less than 200 meters, the distance error threshold is set to 30 centimeters, when the shooting distance is 200-400 meters, the distance error threshold is set to 40 centimeters, and when the shooting distance is greater than 400 meters, the distance error threshold is set to 50 centimeters.

[0063] As another feasible method, the screening parameters may include the number of observation points, and screening the depth information of the depth maps corresponding to multiple perspectives based on the number of observation points includes: spatially aligning the depth maps corresponding to the multiple perspectives, taking each spatial point as a target point, and determining the number of observation points corresponding to the target point based on the depth confidence distribution of the multiple observation points corresponding to the target point, the observation point being the point in the depth map corresponding to the multiple perspectives that is aligned with the target point; for multiple observation points aligned with the same target point, screening the number of observation points corresponding to the target point from the multiple observation points.

[0064] Specifically, for target points with extremely low confidence, they do not participate in point cloud generation because of their low reliability; for target points with higher confidence, these areas require fewer observation points to generate high-quality point clouds, and the number of observation points can be set to 2-3; for target points with lower confidence, these areas have moderate reliability, and the number of observation points can be set to 4-5.

[0065] The above screening parameters given in this application can be used only one for deep information screening, or two or more screening parameters can be used for comprehensive screening. Those skilled in the art can flexibly select according to actual needs.

[0066] In addition to the above-mentioned screening parameters, other screening parameters may also be used. For example, a score is assigned to each depth map based on the estimated quality of the depth map, such as accuracy, completeness, and consistency. The screening parameter may be a quality score threshold for the depth map, and screening is performed based on the score threshold. The depth gradient threshold may also be used as a screening parameter. The gradient size of the depth map may reflect the severity of surface changes, and a threshold may be set to exclude areas with excessively large or small gradients.

[0067] The present application obtains a dense point cloud by fusing the screened depth map. The fusion of the depth map can be achieved through a variety of methods. The screened depth map can be spatially registered by methods such as ICP (Iterative Closest Point) and RANSAC (Random Sample Consensus), so that the depth maps of different perspectives are spatially aligned to ensure their consistency in three-dimensional space; then the depth map interpolation algorithm, weighted average algorithm and other fusion algorithms are used to fuse the registered depth map to obtain a fused depth map. Then, using the camera model parameters and the depth information in the depth map, each pixel in the depth map is converted from the image coordinate system to the three-dimensional space coordinate system, where the depth information actually corresponds to the z-axis in the three-dimensional space coordinate system. Then, after the conversion results of each perspective are registered and aligned, they are merged into point cloud data, i.e., the dense point cloud in the embodiment of the present application.

[0068] The above step 203, i.e., "performing three-dimensional Gaussian splash processing on the point cloud to obtain a three-dimensional real scene model of the target area; wherein the origin of the camera coordinate system used in the three-dimensional Gaussian splash processing is determined according to the camera model parameters" is described in detail below in conjunction with an embodiment.

[0069] This application obtains a three-dimensional real-scene model of the target area through three-dimensional Gaussian Splatting processing on point clouds. Three-dimensional Gaussian Splatting is named because it uses three-dimensional Gaussian distribution as the basic unit of rendering. It is a method for restoring three-dimensional scenes based on three-dimensional Gaussian volumes, and has an excellent new perspective synthesis effect.

[0070] The camera model parameters of satellite images are different from those of traditional pinhole camera models, and the origin of the camera coordinate system is located outside the image range and has a very far pixel position. However, the default camera coordinate system origin position of the traditional three-dimensional Gaussian splash algorithm is the center of the image, which is not suitable for the processing of satellite images. Therefore, the present application modifies the position of the camera coordinate system origin of the three-dimensional Gaussian splash algorithm. The origin of the camera coordinate system is one of the parameters in the camera model parameters. The value of the origin of the camera coordinate system can be determined by the camera model parameters of the target area image obtained in step 202 above, and the parameter value of the origin of the camera coordinate system in the three-dimensional Gaussian splash processing is determined based on the value. Specifically, it can be achieved by modifying the numerical value of the parameter code such as the origin of the camera coordinate system in the three-dimensional Gaussian splash algorithm. This modification supports the free camera coordinate system origin in the three-dimensional Gaussian kernel projection rendering and gradient calculation process, making it possible to train the three-dimensional Gaussian splash algorithm in satellite image scenes.

[0071] Performing three-dimensional Gaussian splashing processing on the point cloud to obtain a three-dimensional real scene model of the target area includes: converting each point in the point cloud into a Gaussian kernel, and the parameters of the Gaussian kernel include the Gaussian distribution, color and opacity corresponding to the point in the point cloud; training the parameters of the Gaussian kernel, and the training goal includes minimizing the difference between the image obtained after projecting the Gaussian kernel corresponding to each point in the point cloud from the camera coordinate system to the pixel coordinate system and the image of the target area; after the training is completed, a three-dimensional real scene model of the target area is obtained.

[0072] The 3D real scene model is essentially composed of a series of Gaussian kernels, which can be mapped to a specific perspective in the subsequent rendering process of the 3D real scene model to generate an image. These Gaussian kernels are represented by Gaussian distribution, color, and opacity, and are usually encoded as vector representations. Then a series of Gaussian kernels are actually encoded as Gaussian distribution matrices, color matrices, and opacity matrices. After these matrices are initialized, they are continuously optimized through a series of training iterations. The resulting Gaussian kernels constitute the 3D real scene model.

[0073] The training goal adopted in the embodiment of the present application is actually to compare the image obtained after projecting each Gaussian kernel at a specific perspective with the target area image corresponding to the specific perspective, and gradually make the two as close as possible. If they are very close, it means that the representation of the Gaussian kernel obtained at this time is already very accurate.

[0074] According to the above training objectives, a loss function is constructed, and the value of the loss function is used in each round of iteration to update the model parameters by methods such as gradient descent until the preset training end conditions are met. The training end conditions may include, for example, the value of the loss function is less than or equal to the preset loss function threshold, the number of iterations reaches the preset number threshold, etc.

[0075] Preferably, the training termination condition includes: the degree of change of the loss function is less than or equal to a preset threshold. Specifically, during the training process, the training is terminated when the difference between the loss function of the current iteration and the loss function of the previous iteration is less than or equal to the preset threshold. In the traditional Gaussian splash algorithm, model training requires 30,000 iterations, and there are fewer satellite image perspectives. Multiple iterations will cause overfitting. Therefore, this method improves the three-dimensional Gaussian splash algorithm based on the characteristics of satellite images, and can terminate the training when the loss function changes little or remains basically unchanged, thereby reducing the burrs and defects caused by overfitting and improving the rendering quality of satellite images. Among them, the preset threshold can be determined based on experience with reference to the number of satellite images, shooting height, etc.

[0076] In the training process of this application, there are few satellite image perspective pictures, poor consistency between views, and the range of variation of the Gaussian kernel parameters is too large, which will cause unreasonable settings of parameters such as the Gaussian kernel position and size, affecting the training effect. Therefore, the training effect can be improved by limiting the step size of the update of the Gaussian kernel parameters. For example, the amount of parameter update in each iteration can be reduced, or a smaller initial learning rate can be set, etc., and regularization terms (such as L1 or L2 regularization) can be added to help control the complexity of the model, indirectly affecting the step size of the parameter update.

[0077] In this application, when the target area image is a plurality of slices, the above-mentioned three-dimensional Gaussian splash processing is performed on the point clouds corresponding to the plurality of slices respectively to obtain the three-dimensional models corresponding to the plurality of switching images, and the three-dimensional models corresponding to the plurality of slices are spliced ​​to obtain the three-dimensional real scene model of the target area. During the splicing process, the overlapping parts between the slices need to be considered, such as using edge detection technology, seamless texture technology and fusion technology, to ensure seamless splicing.

[0078] The above method provided in the embodiment of the present application can be applied to a variety of application scenarios, including but not limited to: application in urban planning and management, restoring actual scenes such as urban buildings, roads and infrastructure through satellite images, which can help planners to carry out urban planning, traffic design and urban landscape design. It can also be applied to the field of environmental monitoring and ecological research, and conduct ecological research and environmental impact assessment by monitoring environmental factors such as terrain changes and vegetation coverage. Three-dimensional reconstruction technology can also provide a more intuitive map display, for example, displaying buildings and terrain through a three-dimensional view to enhance the user's navigation experience.

[0079] It can be seen from the description of the above embodiments that the technical solution provided by the present application can also have the following advantages:

[0080] This application preprocesses and segments the satellite image to obtain the target area image, then obtains the RPC parameters of the remote sensing camera model of the target area image, and uses the RPC parameters to obtain the camera model parameters of the target area image. By preprocessing the satellite image, the quality of the satellite image is improved; by segmenting to obtain the target area image, computing resources are saved; thereby improving the accuracy of the camera model parameters.

[0081] The present application divides the preprocessed image into multiple slices as the target area image, performs three-dimensional Gaussian splash processing on the slices to obtain the corresponding three-dimensional model, and splices the three-dimensional model to obtain a three-dimensional real scene model. The amount of data for each three-dimensional Gaussian splash processing operation can be reduced, thereby speeding up the processing speed, reducing the required memory, and improving efficiency, flexibility and accuracy.

[0082] The camera model parameters of the present application include not only pinhole camera model parameters but also tilt parameters. The tilt parameters provide the precise positional relationship between the images, thus improving the accuracy of the subsequent three-dimensional Gaussian splash processing.

[0083] This application performs feature matching on the target area, and performs motion structure recovery (SFM) based on the matching results and camera model parameters to obtain a sparse point cloud, and uses the sparse point cloud to perform multi-view stereo MVS estimation to generate a dense point cloud. This generates denser point cloud data and improves the quality of point cloud generation.

[0084] In the process of generating a dense point cloud, the present application generates depth maps corresponding to multiple perspectives, fuses the depth maps corresponding to the multiple perspectives, and selects some or all of the points to form a dense point cloud based on the relationship between the depth confidence corresponding to each point in the fused depth map and the confidence threshold. In the process of building a dense point cloud, more accurate depth information is referenced to improve the generation quality of the dense point cloud.

[0085] In the process of 3D Gaussian splashing, this application converts each point in the point cloud into a Gaussian kernel. The training target includes the difference between the image obtained by projecting the Gaussian kernel corresponding to each point from the camera coordinate system to the pixel coordinate system and the image of the target area. In addition, the training end condition and parameter update step size are limited during the training process. The 3D Gaussian splashing algorithm is improved for satellite images, which reduces the burrs and defects caused by overfitting and improves the reconstruction quality of the 3D scene.

[0086] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0087] According to an embodiment of another aspect, a three-dimensional reconstruction apparatus is provided. Figure 4 A schematic block diagram of a three-dimensional reconstruction device according to an embodiment is shown. The device is arranged at Figure 1 The server side of the architecture shown in Figure 1. Figure 4 As shown, the device 400 includes:

[0088] The satellite image acquisition unit 401 is configured to acquire a satellite image containing a target area.

[0089] The target area acquisition unit 402 is configured to obtain a target area image and camera model parameters corresponding to the target area image using satellite images.

[0090] The point cloud construction unit 403 is configured to construct a point cloud using the target area image and the camera model parameters corresponding to the target area image.

[0091] The Gaussian reconstruction unit 404 is configured to perform three-dimensional Gaussian splash processing on the point cloud to obtain a three-dimensional real scene model of the target area; wherein the origin of the camera coordinate system used in the three-dimensional Gaussian splash processing is determined according to the camera model parameters.

[0092] As one of the feasible ways, when the target area acquisition unit 402 obtains the target area image and the camera model parameters of the target area image using the satellite image, it can be specifically configured as follows: preprocessing the satellite image to obtain the preprocessed image; segmenting the preprocessed image according to the range of the target area to obtain the target area image; obtaining the rational polynomial coefficients RPC parameters corresponding to the target area image; and obtaining the camera model parameters of the target area image using the RPC parameters.

[0093] As one of the feasible methods, when the target area acquisition unit 402 divides the preprocessed image according to the range of the target area to obtain the target area image, it can be configured as follows: divide the preprocessed image within the range of the target area to obtain multiple slice images, and use the multiple slice images as the target area image; when the Gaussian reconstruction unit 404 performs three-dimensional Gaussian splash processing on the point cloud to obtain the three-dimensional real scene model of the target area, it can be configured as follows: perform three-dimensional Gaussian splash processing on the point clouds corresponding to the multiple slice images respectively to obtain three-dimensional models corresponding to the multiple slice images, and splice the three-dimensional models corresponding to the multiple slice images to obtain the three-dimensional real scene model of the target area.

[0094] As one of the feasible ways, the target area acquisition unit 402 can be configured as follows: preprocessing includes at least one of the following: radiation correction of satellite images; dehazing of satellite images; geometric correction of satellite images; full-color fusion of satellite images; and super-resolution processing of satellite images.

[0095] As one achievable manner, the point cloud construction unit 403 may be configured as follows: the camera model parameters include pinhole camera model parameters and tilt parameters.

[0096] As one of the feasible ways, when the Gaussian reconstruction unit 404 performs three-dimensional Gaussian splash processing on the point cloud to obtain a three-dimensional real scene model of the target area, it can be configured as follows: each point in the point cloud is converted into a Gaussian kernel, and the parameters of the Gaussian kernel include the Gaussian distribution, color and opacity corresponding to the point in the point cloud; the parameters of the Gaussian kernel are trained, and the training goal includes minimizing the difference between the image obtained after projecting the Gaussian kernel corresponding to each point in the point cloud from the camera coordinate system to the pixel coordinate system and the image of the target area; after the training is completed, a three-dimensional real scene model of the target area is obtained.

[0097] As one of the achievable ways, the Gaussian reconstruction unit 404 may be configured as follows: the end condition of the training includes: the degree of change of the loss function is less than or equal to a preset threshold, and the loss function is obtained according to the training goal.

[0098] As one achievable manner, the Gaussian reconstruction unit 404 may be configured to reduce the update step size of the Gaussian kernel parameters during the training process.

[0099] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0100] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0101] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0102] And an electronic device, comprising: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, and the program instructions, when read and executed by the one or more processors, execute the steps of any of the methods described in the aforementioned method embodiments.

[0103] The present application also provides a computer program product, including a computer program, which implements the steps of any one of the methods in the aforementioned method embodiments when executed by a processor.

[0104] in, Figure 5 The architecture of the electronic device is shown as an example, which may include a processor 510, a video display adapter 511, a disk drive 512, an input / output interface 513, a network interface 514, and a memory 520. The processor 510, the video display adapter 511, the disk drive 512, the input / output interface 513, the network interface 514, and the memory 520 may be communicatively connected via a communication bus 530.

[0105] The processor 510 may be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided in this application.

[0106] The memory 520 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 520 can store an operating system 521 for controlling the operation of the electronic device 500, and a basic input and output system (BIOS) 522 for controlling the low-level operation of the electronic device 500. In addition, a web browser 523, a data storage management system 524, and a three-dimensional reconstruction device 525, etc. can also be stored. The above-mentioned three-dimensional reconstruction device 525 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 520 and is called and executed by the processor 510.

[0107] The input / output interface 513 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0108] The network interface 514 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0109] The bus 530 comprises a pathway for transmitting information between the various components of the device (eg, the processor 510, the video display adapter 511, the disk drive 512, the input / output interface 513, the network interface 514, and the memory 520).

[0110] It should be noted that, although the above device only shows a processor 510, a video display adapter 511, a disk drive 512, an input / output interface 513, a network interface 514, a memory 520, a bus 530, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0111] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0112] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A three-dimensional reconstruction method, characterized in that: The method comprises: Acquire satellite images containing the target area; Obtaining a target area image and camera model parameters corresponding to the target area image using the satellite image; Constructing a point cloud using the target area image and camera model parameters corresponding to the target area image; The point cloud is subjected to three-dimensional Gaussian splash processing to obtain a three-dimensional real scene model of the target area; wherein the origin of the camera coordinate system used in the three-dimensional Gaussian splash processing is determined according to the camera model parameters.

2. The method according to claim 1, characterized in that: The step of obtaining a target area image and camera model parameters corresponding to the target area image by using the satellite image includes: Preprocessing the satellite image to obtain a preprocessed image; Segment the preprocessed image according to the range of the target area to obtain a target area image; Obtaining the rational polynomial coefficient RPC parameters corresponding to the target area image; The RPC parameters are used to obtain the camera model parameters corresponding to the target area image.

3. The method according to claim 2, characterized in that The step of segmenting the preprocessed image according to the range of the target area to obtain the target area image includes: Slicing the preprocessed image within the range of the target area to obtain a plurality of slice images, and using the plurality of slice images as the target area image; The step of performing three-dimensional Gaussian splash processing on the point cloud to obtain a three-dimensional real scene model of the target area includes: The point clouds corresponding to the multiple slice images are respectively subjected to three-dimensional Gaussian splash processing to obtain three-dimensional models corresponding to the multiple slice images, and the three-dimensional models corresponding to the multiple slice images are spliced ​​to obtain a three-dimensional real scene model of the target area.

4. The method according to claim 2, characterized in that: The pretreatment includes at least one of the following: performing radiometric correction on the satellite image; Dehazing the satellite image; performing geometric correction on the satellite image; Performing full-color fusion on the satellite image; The satellite image is subjected to super-resolution processing.

5. The method according to claim 1, characterized in that The camera model parameters include pinhole camera model parameters and tilt parameters.

6. The method according to any one of claims 1 to 5, characterized in that The step of performing three-dimensional Gaussian splash processing on the point cloud to obtain a three-dimensional real scene model of the target area includes: Convert each point in the point cloud into a Gaussian kernel, wherein the parameters of the Gaussian kernel include the Gaussian distribution, color and opacity corresponding to the point in the point cloud; Training the parameters of the Gaussian kernel, wherein the training objective includes minimizing the difference between an image obtained by projecting the Gaussian kernel corresponding to each point in the point cloud from a camera coordinate system to a pixel coordinate system and the image of the target area; After the training is completed, a three-dimensional real scene model of the target area is obtained.

7. The method according to claim 6, characterized in that The end conditions of the training include: The degree of change of the loss function is less than or equal to a preset threshold, and the loss function is obtained according to the training objective.

8. The method according to claim 6, characterized in that During the training process, the update step size of the parameters of the Gaussian kernel is reduced.

9. A three-dimensional reconstruction device, characterized in that: The device comprises: A satellite image acquisition unit, which acquires a satellite image containing a target area; A target area acquisition unit, which uses the satellite image to obtain a target area image and camera model parameters corresponding to the target area image; A point cloud construction unit, which constructs a point cloud using the target area image and camera model parameters corresponding to the target area image; A Gaussian reconstruction unit performs a three-dimensional Gaussian splashing process on the point cloud to obtain a three-dimensional real scene model of the target area; wherein the origin of the camera coordinate system used in the three-dimensional Gaussian splashing process is determined according to the camera model parameters.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

11. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of claims 1 to 8.

Images