Three-dimensional reconstruction method and corresponding device

Abstract

The embodiment of the invention discloses a three-dimensional reconstruction method and a corresponding device. The three-dimensional reconstruction method comprises the following steps: acquiring target area images of a plurality of view angles; generating dense point clouds by using the target area images of the multiple visual angles; performing Gaussian splashing processing on the dense point cloud to obtain a three-dimensional model of the target area; wherein part of Gaussian kernels are deleted according to the importance degree of each Gaussian kernel in the Gaussian splashing treatment, and the importance degree of the Gaussian kernels is determined at least based on the opacity of the Gaussian kernels. According to the method, part of Gaussian kernels are deleted according to the importance degree of the Gaussian kernels in Gaussian splashing processing, and the operation cost is saved under the condition that the three-dimensional reconstruction quality is guaranteed.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a three-dimensional reconstruction method and device. Background Art

[0002] 3D reconstruction technology has been widely used in virtual reality, augmented reality and game development, architecture, urban planning and Geographic Information System (GIS). Point cloud is a data structure composed of a large number of points, which is usually used to represent objects or scenes in three-dimensional space.

[0003] Traditional 3D reconstruction methods usually shoot the same object from multiple perspectives, then generate point clouds through motion recovery structure processing, and then perform 3D reconstruction from the point clouds. Since 3D reconstruction relies on matching calculations of a large amount of point cloud data from multiple perspectives, traditional 3D reconstruction methods require high computing costs while ensuring the quality of 3D reconstruction. Summary of the invention

[0004] The present application provides a three-dimensional reconstruction method and a corresponding device, so as to save computing costs while ensuring the quality of three-dimensional reconstruction.

[0005] This application provides the following solutions:

[0006] According to a first aspect, a three-dimensional reconstruction method is provided, the method comprising: acquiring target area images from multiple perspectives; generating a dense point cloud using the target area images from multiple perspectives; performing Gaussian splattering on the dense point cloud to obtain a three-dimensional model of the target area; wherein, in the Gaussian splattering process, some Gaussian kernels are deleted according to the importance of each Gaussian kernel, and the importance of the Gaussian kernel is determined at least based on the opacity of the Gaussian kernel.

[0007] According to the second aspect, a three-dimensional reconstruction device is provided, which includes: an image acquisition unit, configured to acquire target area images from multiple perspectives; a point cloud generation unit, configured to generate a dense point cloud using the target area images from multiple perspectives; a Gaussian splash unit, configured to perform Gaussian splash processing on the dense point cloud to obtain a three-dimensional model of the target area; wherein, in the Gaussian splash processing, some Gaussian kernels are deleted according to the importance of each Gaussian kernel, and the importance of the Gaussian kernel is determined at least based on the opacity of the Gaussian kernel.

[0008] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] According to a fourth aspect, an electronic device is provided, comprising: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, and the program instructions, when read and executed by the one or more processors, execute the steps of the method described in the first aspect above.

[0010] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0011] In the process of Gaussian splattering of dense point clouds, the present application deletes some Gaussian kernels according to their importance, and the importance of the Gaussian kernel is determined at least based on the opacity of the Gaussian kernel. Since the opacity of the Gaussian kernel is related to the contribution of the rendering in the three-dimensional reconstruction, the importance of the Gaussian kernel is determined based on the opacity of the Gaussian kernel, thereby deleting some Gaussian kernels, which can save computing costs while ensuring the quality of the three-dimensional reconstruction.

[0012] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 A system architecture diagram applicable to the embodiments of the present application;

[0015] Figure 2 A flowchart of a three-dimensional reconstruction method provided in an embodiment of the present application;

[0016] Figure 3 A depth information estimation status diagram provided in an embodiment of the present application;

[0017] Figure 4 A schematic diagram of the process of deep information screening provided in an embodiment of the present application;

[0018] Figure 5 A structural block diagram of a three-dimensional reconstruction device provided in an embodiment of the present application;

[0019] Figure 6 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0021] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0022] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0023] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0024] There are already some 3D reconstruction technologies, which are usually based on point cloud Gaussian splatter processing to obtain a 3D reconstruction model. These technologies render all Gaussian kernels converted from dense point clouds. However, rendering some less important Gaussian kernels not only has little improvement on the 3D reconstruction effect, but also takes up computing resources.

[0025] In view of this, the present application provides a new idea. In order to facilitate the understanding of the present application, the system architecture on which the present application is based is first described. Figure 1 An exemplary system architecture to which the embodiments of the present application can be applied is shown. Figure 1 As shown in , the system architecture may include: user equipment and a three-dimensional reconstruction device located at the map application server end.

[0026] The user can input a three-dimensional reconstruction request through the user device, and the user device sends the request to the three-dimensional reconstruction device of the map application server. The three-dimensional reconstruction device can use the method provided in the embodiment of the present application to perform three-dimensional reconstruction on the target area in the target area image to obtain a three-dimensional real scene model. The server side of the map application can respond to the request of the user terminal and send the three-dimensional real scene model of the target area to the user terminal, and the user terminal uses the three-dimensional real scene model for rendering to obtain a two-dimensional image, a three-dimensional image, a VR (Virtual Reality) scene or an AR (Augmented Reality) scene, etc.

[0027] Among them, user equipment may include but is not limited to: smart mobile terminals, smart home devices, wearable devices, PC (Personal Computer), etc. Among them, smart mobile devices may include such as mobile phones, tablet computers, laptops, PDAs (Personal Digital Assistants), Internet cars, etc. Smart home devices may include smart TVs, smart refrigerators, etc. Wearable devices may include such as smart watches, smart glasses, virtual reality devices, augmented reality devices, mixed reality devices (i.e., devices that can support virtual reality and augmented reality), etc.

[0028] The 3D reconstruction device can be set up as an independent server, a server group, or a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product in the cloud computing service system to solve the problems of difficult management and weak service scalability in traditional physical hosts and virtual private servers (VPS). Figure 1 In addition to the illustrated architecture, the 3D reconstruction device may also be disposed on a computer terminal with relatively strong computing capabilities.

[0029] It should be understood that Figure 1 The user equipment and the 3D reconstruction device in the figure are only for illustration. Any number of user equipment and 3D reconstruction devices may be provided according to implementation requirements.

[0030] Figure 2 A flowchart of a three-dimensional reconstruction method provided in an embodiment of the present application. The method can be performed by Figure 1 The three-dimensional reconstruction device in the system shown is performed. Figure 2 As shown in , the method may include the following steps:

[0031] Step 201: Acquire target area images from multiple viewing angles.

[0032] Step 202: Generate a dense point cloud using target area images from multiple perspectives.

[0033] Step 203, performing Gaussian splash processing on the dense point cloud to obtain a three-dimensional model of the target area; wherein, in the Gaussian splash processing, some Gaussian kernels are deleted according to the importance of each Gaussian kernel, and the importance of the Gaussian kernel is determined at least based on the opacity of the Gaussian kernel.

[0034] It can be seen from the above process that in the process of Gaussian splattering of dense point clouds, the present application deletes some Gaussian kernels according to the importance of each Gaussian kernel, and the importance of the Gaussian kernel is at least determined based on the opacity of the Gaussian kernel. Since the opacity of the Gaussian kernel is related to the contribution of the rendering in the three-dimensional reconstruction, by determining the importance of the Gaussian kernel based on the opacity of the Gaussian kernel and deleting some Gaussian kernels, the computing cost can be saved while ensuring the quality of the three-dimensional reconstruction.

[0035] The following describes in detail each step in the above process and the effects that can be further produced in conjunction with the embodiments.

[0036] First, the above step 201, namely "obtaining target area images from multiple perspectives", is described in detail in conjunction with the embodiments.

[0037] The present application aims to perform three-dimensional reconstruction on a target area image, where the target area image is an image containing a target area. The present application obtains multiple target area images, wherein one target area image may contain the entire target area, or may only contain a portion of the target area. The target area image can be obtained in a variety of ways, for example, through public data platforms such as Pexels, Unsplash, and Pixabay, which provide a large number of public domain aerial images. It can also be obtained by photographing the ground target area through image acquisition devices such as drones or manned aircraft. After the image containing the target area is obtained by the data platform or the shooting device, the target area image can be obtained through pre-processing operations such as shearing and geometric correction.

[0038] In order to ensure the quality of 3D reconstruction, the target area image contains multiple viewing angles, and provides multi-view information through different imaging angles. The image acquisition device can shoot the target area at different imaging angles.

[0039] The above step 202, namely "generating a dense point cloud using target area images from multiple perspectives", is described in detail below in conjunction with an embodiment.

[0040] There are many ways to generate dense point clouds using images of the target area from multiple perspectives. For example, using image data from multiple perspectives to generate point clouds through a stereo matching algorithm; or using a deep learning model to generate a three-dimensional point cloud from a two-dimensional image.

[0041] As an implementable method, Structure from Motion (SFM) processing can be performed on the target area images of multiple perspectives to obtain the camera parameters and pose information corresponding to each target image. Multi-view stereo vision (MVS) estimation is performed using the camera parameters and pose information to obtain depth maps corresponding to multiple perspectives. The depth information of the depth maps corresponding to multiple perspectives is filtered according to the filtering parameters, where the filtering parameters are determined according to the depth information estimation status corresponding to each depth map. The filtered depth maps are fused to obtain a dense point cloud.

[0042] SFM is a computer vision technology used to recover the 3D structure of a scene and the motion state of the camera from a set of static images without prior information. It infers the motion trajectory of the camera and the 3D structure of the scene by analyzing the similarities between images.

[0043] Camera parameters refer to the internal parameters of the camera, including focal length, principal point position, pixel size, and lens distortion coefficient. They define the imaging characteristics of the camera, such as the viewing angle, shape, and distortion of the image. The pose information describes the position (represented by a translation vector) and direction (represented by a rotation matrix or Euler angle) of the camera in three-dimensional space. The pose information can be used to obtain information such as the cruising altitude and position of the aircraft, and the pitch angle of the aircraft.

[0044] SFM technology can obtain the camera parameters and pose information corresponding to each target image by performing feature matching on the target area images corresponding to multiple perspectives. Among them, incremental SFM can be used, starting from an initial image pair, and gradually adding new images to the reconstruction; global SFM can also be used to process all input images at the same time, and build a model of the entire scene by jointly optimizing the camera poses and three-dimensional point positions of all images; deep learning models can also be used to extract feature points and then perform SFM. Preferably, software tools such as 3D reconstruction software colmap and OpenMVG can be used to perform SFM on the target area images to obtain the camera parameters and pose information corresponding to each target image, providing a basis for subsequent 3D modeling.

[0045] MVS is also a computer vision technology used to calculate the three-dimensional structure of a scene from multiple images taken from different perspectives. MVS mainly estimates the depth information of the same object or scene element by comparing it from different perspectives, thereby generating point cloud data or a three-dimensional mesh model. The core idea of ​​this technology is to use the parallax between multiple images to infer the depth information of the object, and then construct a three-dimensional model of the scene.

[0046] Multi-view stereo MVS estimation using camera parameters and pose information can be achieved through a variety of methods. For example, a multi-view stereo vision algorithm is used to construct matching costs using geometric and optical consistency, accumulate matching costs, and then estimate depth values.

[0047] As an implementable method, MVS estimation can be performed on multi-view target images by combining camera parameters and pose information through tools such as colmap or GeoMVSNet, thereby generating depth maps corresponding to multiple viewpoints. The depth map contains the depth information of each pixel, where the depth information includes depth value and depth confidence, and the depth confidence is information that characterizes the credibility of the depth value.

[0048] Specifically, taking the use of GeoMVSNet for MVS estimation as an example, Figure 3 The depth map example provided in the embodiment of the present application is as follows: Figure 3 As shown, Figure 3 (a) is the target area image input to GeoMVSNet. Figure 3 (b) is a visualized depth map generated by GeoMVSNet for the target area image. The colors in the map represent the depth values ​​corresponding to each pixel. The darker the blue, the larger the depth value, and the yellow, the smaller the depth value. Figure 3 (c) is a visualized confidence map generated by GeoMVSNet for the target area image. The colors in the map represent the depth confidence of each pixel. The darker the blue, the greater the confidence value, and the yellow represents the smaller the depth confidence value. Figure 3 (d) is a depth probability distribution diagram, where the horizontal axis is the depth value and the vertical axis is the number of pixels corresponding to the depth value; Figure 3 (e) is a confidence probability distribution diagram, where the horizontal axis is the depth confidence value and the vertical axis is the number of pixels corresponding to the depth confidence value.

[0049] The present application screens depth information of depth maps corresponding to multiple perspectives based on screening parameters, and the screening parameters are determined based on the depth information estimation status corresponding to each depth map. Figure 4 A schematic diagram of the process of deep information screening provided in the embodiment of the present application is shown in FIG. Figure 4As shown, after obtaining the depth information corresponding to the depth map, after determining the screening parameters according to the depth information, the other depth maps are projected onto the current depth map, and then the depth information is screened. After traversing all the depth maps, the depth information screening result is obtained.

[0050] As an implementable method, the screening parameters may include a confidence threshold, and screening the depth information of the depth maps corresponding to multiple perspectives based on the confidence threshold includes: determining the confidence threshold of each depth map based on the depth confidence distribution corresponding to each depth map; filtering out the depth information in each depth map whose confidence is less than or equal to the corresponding confidence threshold.

[0051] The confidence threshold is determined based on the depth confidence distribution corresponding to each depth map. Specifically, the statistical data (such as the mean, median or standard deviation) of the depth confidence distribution can be calculated, and the confidence threshold can be determined using these statistical data. Preferably, the confidence threshold of the depth map can be determined based on the depth confidence corresponding to the probability peak in the probability distribution of the depth confidence of the depth map. Figure 3 As shown in (e), in the probability distribution of depth confidence, the confidence corresponding to the peak is usually the depth confidence corresponding to the largest number of pixels in the overall distribution, and the depth confidence can be used as the confidence threshold of the depth map.

[0052] As another practicable manner, the screening parameter may include a distance error threshold, and the distance error threshold of each depth map is determined based on the shooting distance of the target area image to which each depth map belongs.

[0053] For the depth information of pixel point i on the depth map corresponding to viewing angle a, if the distance between the depth information of pixel point i obtained by mapping the depth maps corresponding to other viewing angles to viewing angle a exceeds the distance error threshold of the depth map corresponding to viewing angle a, indicating that the depth information error of pixel point i is relatively large, then the depth information of pixel point i on the depth map corresponding to viewing angle a is filtered out, viewing angle a is any viewing angle among multiple viewing angles, and pixel point i is any point on the target area image corresponding to viewing angle a.

[0054] The distance error threshold of the depth map is positively correlated with the shooting distance of the target area image to which the depth map belongs. The positive correlation can be linear or nonlinear. For example, when the shooting distance is less than 200 meters, the distance error threshold is set to 30 centimeters, when the shooting distance is 200-400 meters, the distance error threshold is set to 40 centimeters, and when the shooting distance is greater than 400 meters, the distance error threshold is set to 50 centimeters.

[0055] As another feasible method, the screening parameters may include the number of observation points, and screening the depth information of the depth maps corresponding to multiple perspectives based on the number of observation points includes: spatially aligning the depth maps corresponding to the multiple perspectives, taking each spatial point as a target point, and determining the number of observation points corresponding to the target point based on the depth confidence distribution of the multiple observation points corresponding to the target point, the observation point being the point in the depth map corresponding to the multiple perspectives that is aligned with the target point; for multiple observation points aligned with the same target point, screening the number of observation points corresponding to the target point from the multiple observation points.

[0056] Specifically, for target points with extremely low confidence, such as Figure 3 The yellow area shown in (c) does not participate in point cloud generation because the reliability of these target points is low; for target points with higher confidence, such as Figure 3 The blue areas shown in (c) require fewer observation points to generate high-quality point clouds, and the number of observation points can be set to 2-3; for target points with lower confidence, such as Figure 3 (c) shows the areas between yellow and blue. These areas have moderate reliability and the number of observation points can be set to 4-5.

[0057] The above screening parameters given in this application can be used only one for deep information screening, or two or more screening parameters can be used for comprehensive screening. Those skilled in the art can flexibly select according to actual needs.

[0058] In addition to the above-mentioned screening parameters, other screening parameters may also be used. For example, a score is assigned to each depth map based on the estimated quality of the depth map, such as accuracy, completeness, and consistency. The screening parameter may be a quality score threshold for the depth map, and screening is performed based on the score threshold. The depth gradient threshold may also be used as a screening parameter. The gradient size of the depth map may reflect the severity of surface changes, and a threshold may be set to exclude areas with excessively large or small gradients.

[0059] This application obtains a dense point cloud by fusing the screened depth map. The fusion of depth maps can be achieved through a variety of methods. The screened depth map can be spatially aligned by methods such as ICP (Iterative Closest Point) and RANSAC (Random Sample Consensus), so that the depth maps of different perspectives are spatially aligned to ensure their consistency in three-dimensional space; then the depth map interpolation algorithm, weighted average algorithm and other fusion algorithms are used to fuse the registered depth maps to obtain a fused depth map. Then, using the camera model parameters and the depth information in the depth map, each pixel in the depth map is converted from the image coordinate system to the three-dimensional space coordinate system to obtain a dense point cloud. The depth information actually corresponds to the z-axis in the three-dimensional space coordinate system.

[0060] After obtaining the dense point cloud, the generated dense point cloud may be further subjected to post-processing such as denoising, smoothing, filtering, etc., which will not be described in detail in the embodiments of the present application.

[0061] The above step 203, i.e. "performing Gaussian splash processing on the dense point cloud to obtain a three-dimensional model of the target area; wherein, in the Gaussian splash processing, some Gaussian kernels are deleted according to the importance of each Gaussian kernel, and the importance of the Gaussian kernel is determined at least based on the opacity of the Gaussian kernel." is described in detail in conjunction with the embodiments.

[0062] After the dense point cloud is generated by the point cloud generation method given in the above embodiment, the dense point cloud is processed by 3D Gaussian Splatting to obtain a 3D model of the target area.

[0063] The three-dimensional Gaussian splash processing is named because it uses three-dimensional Gaussian distribution as the basic unit of rendering. It is a method of restoring three-dimensional scenes based on three-dimensional Gaussian volumes, and has a good new perspective synthesis effect. Gaussian splash processing is performed on dense point clouds to obtain a three-dimensional model of the target area, including: converting each point in the dense point cloud into a Gaussian kernel, and the parameters of the Gaussian kernel include the Gaussian distribution representation, color representation and opacity representation corresponding to the points in the dense point cloud. The parameters of the Gaussian kernel are trained, and the training objectives include: minimizing the difference between the image rendered by the Gaussian kernel corresponding to each point in the dense point cloud and the image of the target area. After the training is completed, a three-dimensional real-scene model of the target area is obtained.

[0064] The 3D real scene model is essentially composed of a series of Gaussian kernels, which can be mapped to a specific perspective in the subsequent rendering process of the 3D real scene model to generate an image. These Gaussian kernels are represented by Gaussian distribution, color, and opacity, and are usually encoded as vector representations. Then a series of Gaussian kernels are actually encoded as Gaussian distribution matrices, color matrices, and opacity matrices. After these matrices are initialized, they are continuously optimized through a series of training iterations. The resulting Gaussian kernels constitute the 3D real scene model.

[0065] The training goal adopted in the embodiment of the present application is actually to compare the image obtained after projecting each Gaussian kernel at a specific perspective with the target area image corresponding to the specific perspective, and gradually make the two as close as possible. If they are very close, it means that the representation of the Gaussian kernel obtained at this time is already very accurate.

[0066] According to the above training objectives, a loss function is constructed, and the value of the loss function is used in each round of iteration to update the model parameters by methods such as gradient descent until the preset training end conditions are met. The training end conditions may include, for example, the value of the loss function is less than or equal to the preset loss function threshold, the number of iterations reaches the preset number threshold, etc.

[0067] This application proposes a point reduction method for dense point clouds to achieve high-quality representation of three-dimensional scenes with as few points as possible, which is extremely important for real-time rendering on the terminal. Specifically, after training or during training, some Gaussian kernels are deleted according to the importance of each Gaussian kernel, and the importance of the Gaussian kernel is determined at least based on the opacity representation of the Gaussian kernel.

[0068] The opacity representation of the Gaussian kernel can be represented by a variety of parameters, such as the color, opacity, and transmittance of the Gaussian kernel. In order to automatically find a suitable pruning ratio during model training and balance point reduction and quality, a mask value can be learned for each Gaussian kernel, and the mask value can be used to represent the opacity of the Gaussian kernel. When this mask value is applied to model rendering, the closer the mask value is to 0, the smaller the contribution of the corresponding Gaussian point to the rendering.

[0069] As an implementable method, the opacity representation can be obtained based on the opacity and the mask value. At this time, the training goal also includes: maximizing the number of Gaussian kernels whose mask values ​​meet the preset conditions. After the training is completed, based on the mask values ​​of each Gaussian kernel obtained through training, the Gaussian kernels whose mask values ​​meet the preset conditions are deleted.

[0070] Among them, the preset conditions can be set flexibly. For example, the preset condition can be that the mask value is 0, that is, the Gaussian kernel with a mask value of 0 is deleted; the preset condition can also be that the mask value is less than or equal to the mask value threshold, that is, the Gaussian kernel with a smaller mask value is deleted; the Gaussian kernels can also be sorted according to the mask value, and the preset condition is the first several Gaussian kernels with the smallest mask value, that is, the first several Gaussian kernels with the smallest mask value can be deleted.

[0071] The importance of the Gaussian kernel can be determined based on the opacity of the Gaussian kernel. As one of the feasible ways, the importance of the Gaussian kernel can be determined by using the opacity of the Gaussian kernel and the transmittance corresponding to the Gaussian kernel. The transmittance corresponding to the Gaussian kernel refers to the transmittance between the Gaussian kernel and the rendered pixel. For example, for all pixels covered by the Gaussian kernel (i.e., all pixels mapped to the Gaussian kernel during rendering), the product of the opacity (essentially the opacity of the Gaussian kernel) and the transmittance corresponding to each pixel can be determined, and the maximum value of the product is used as the importance of the Gaussian kernel.

[0072] The importance of the Gaussian kernel can also be determined based on the opacity of the Gaussian kernel and the size of the Gaussian kernel, where the size of the Gaussian kernel is related to the standard deviation of the Gaussian distribution. For example, the importance of the Gaussian kernel is positively correlated with the opacity of the Gaussian kernel and the size of the Gaussian kernel. The importance of the Gaussian kernel can be represented by constructing an importance score. The importance of the Gaussian kernel represents the contribution of the Gaussian kernel to the rendering. The smaller the contribution, the weaker its impact on the rendering result, and it can be given priority in the point reduction process.

[0073] In practical applications, the number of points after point reduction may still exceed the target number of points. For example, in scenes rendered on mobile terminals, the number of points of the Gaussian model is required to be extremely small due to the computing power of the device. Therefore, a progressive multi-round point reduction scheme can be adopted to restore the three-dimensional real scene as accurately as possible while reducing the number of points to the limit, and at the same time, the number of points of the final output model can be accurately controlled. Specifically, in each round of training iteration, some Gaussian kernels are deleted according to the importance of each Gaussian kernel so that the number of Gaussian kernels retained after the training is completed meets the preset number requirements; wherein the preset number requirements are determined by the three-dimensional reconstruction quality requirements, and can also be determined according to the computing power of the rendering terminal.

[0074] As one of the feasible methods, the target number of Gaussian cores can be determined based on the computing power of the mobile terminal that performs rendering, and then the current number of Gaussian cores can be reduced to the target number of Gaussian cores within the training rounds based on the number of training rounds of the Gaussian model. The number of Gaussian cores that need to be deleted in each round of iteration is then deleted in each round of iteration based on the importance of each Gaussian core.

[0075] In this application, there may be interference objects in the target area image. Since the interference objects often present different postures under different observation angles, the reconstruction algorithm will overfit the interference objects as "floating objects". In order to alleviate the impact of interference objects, traditional three-dimensional reconstruction algorithms usually need to apply image segmentation technology to generate a binary mask of interference objects for each collected image, so as to avoid reconstruction errors caused by interference objects in subsequent training. However, this method not only has strict requirements on the accuracy of the image segmentation scheme, but also needs to enumerate a large number of categories of interference objects in advance. In addition, when some instances appear to be stationary (such as a car stopped on the road), this segmentation scheme will still segment them as interference objects, resulting in a "hollow" reconstruction effect.

[0076] The traditional Gaussian splash algorithm will first fit the low-frequency signal and then fit the high-frequency signal during the training process. However, dynamic interference objects are not consistent at different viewing angles and can be approximately considered as a kind of high-frequency noise. For this reason, the present application has made modifications to the traditional Gaussian splash algorithm. Specifically, during the first N rounds of training iterations, the present application performs frequency domain analysis and high-frequency noise filtering on the loss function, and then uses the loss function after high-frequency noise filtering to iteratively update the parameters of the Gaussian kernel. The loss function is determined based on the training objectives, and N is less than or equal to the total number of training rounds. By performing frequency domain analysis and high-frequency noise filtering on the loss function during the first N rounds of iterations, the removal of interference objects in the three-dimensional reconstruction process is achieved.

[0077] The target area image may also show inconsistent light colors due to factors such as acquisition time, weather conditions, and post-processing algorithms. In the traditional Gaussian splash algorithm, such inconsistent target area images often cause the reconstructed three-dimensional model to produce undesirable effects such as ghosting and artifacts. In order to solve this problem, the present application introduces picture-level color encoding in the Gaussian splash algorithm, further decomposes the color of the rendered image into scattered light that conforms to Lambertian reflection, and performs inconsistent image affine transformation between frames, thereby improving the color expression ability of the algorithm.

[0078] Specifically, when the Gaussian kernel is rendered to obtain an image, the color of the pixel in the rendered image is affine transformed according to the affine matrix, wherein the affine matrix is ​​updated during the training process. The size of the affine matrix can be flexibly set. Preferably, a learnable affine matrix of size 4*4 can be used. After obtaining the rendering image of the conventional process, the affine transformation is applied to the color channel to approximate the post-processing algorithm of the fitting camera, and the final rendering image is obtained for the calculation of the reconstruction error.

[0079] The above method provided in the embodiment of the present application can be applied to a variety of application scenarios, including but not limited to: application in urban planning and management, by restoring actual scenes such as urban buildings, roads and infrastructure through target area images, it can help planners to carry out urban planning, traffic design and urban landscape design. It can also be applied to the field of environmental monitoring and ecological research, by monitoring environmental factors such as terrain changes and vegetation coverage, to conduct ecological research and environmental impact assessment. Three-dimensional reconstruction technology can also provide a more intuitive map display, for example, by displaying buildings and terrain through a three-dimensional view, enhancing the user's navigation experience.

[0080] It can be seen from the description of the above embodiments that the technical solution provided by the present application can also have the following advantages:

[0081] 1) In the first N iterations of the training process of the 3D Gaussian splatting process, the present application performs frequency domain analysis and high-frequency noise filtering on the loss function, and iteratively updates the parameters of the Gaussian kernel using the loss function after high-frequency noise filtering. This makes the Gaussian training process unaffected by the noise caused by the interference in the target image, thereby improving the quality of 3D reconstruction.

[0082] 2) When rendering an image with a Gaussian kernel, the present application performs an affine transformation on the color of the pixel points in the rendered image, thereby improving the color expression capability of the Gaussian splash algorithm, reducing the impact of inconsistent light color of the target area image on the algorithm, and improving the quality of 3D reconstruction.

[0083] 3) After the training is completed or during the training process, the present application deletes some Gaussian kernels according to their importance. By learning a mask value for each Gaussian kernel and deleting the Gaussian kernels whose mask values ​​meet the preset conditions, a nearly lossless point reduction effect can be achieved, saving computational costs while ensuring the quality of 3D reconstruction.

[0084] 4) In each round of training iteration, the present application deletes some Gaussian kernels according to their importance so that the number of Gaussian kernels retained after the training meets the preset number requirements. The number of points in the point cloud can be accurately controlled. While meeting the preset point number requirements, the most effective points for scene rendering can be retained as much as possible, thereby ensuring the high quality of the final rendering result.

[0085] 5) This application uses a confidence threshold to screen the depth information of the depth map, filters out the depth information with low confidence in the depth map, and eliminates the depth data with high uncertainty, thereby improving the quality of the point cloud.

[0086] 6) This application uses a distance error threshold to filter the depth information of the depth map, following the principle of "the lower the shooting distance, the smaller the allowable distance error", ensuring high standards for depth information during low-altitude shooting, thereby improving the quality of the point cloud.

[0087] 7) This application screens the depth information of the depth map by the number of observation points, and uses different numbers of observation points for target points with different confidence levels, fully considering the breadth and intensity of the confidence distribution, saving computing costs while ensuring the quality of point cloud generation.

[0088] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0089] According to an embodiment of another aspect, a three-dimensional reconstruction apparatus is provided. Figure 5 A schematic block diagram of a three-dimensional reconstruction device according to an embodiment is shown. The device is arranged at Figure 1 The server side of the architecture shown in Figure 1. Figure 5 As shown, the device 500 includes:

[0090] The image acquisition unit 501 is configured to acquire target area images from multiple perspectives.

[0091] The point cloud generating unit 502 is configured to generate a dense point cloud using target area images from multiple perspectives.

[0092] The Gaussian splash unit 503 is configured to perform Gaussian splash processing on the dense point cloud to obtain a three-dimensional model of the target area; wherein, in the Gaussian splash processing, some Gaussian kernels are deleted according to the importance of each Gaussian kernel, and the importance of the Gaussian kernel is determined at least based on the opacity of the Gaussian kernel.

[0093] As one of the feasible ways, when the Gaussian splash unit 503 performs Gaussian splash processing on a dense point cloud to obtain a three-dimensional model of the target area, it can be configured as follows: converting each point in the dense point cloud into a Gaussian kernel, and the parameters of the Gaussian kernel include the Gaussian distribution representation, color representation and opacity representation corresponding to the points in the dense point cloud; training the parameters of the Gaussian kernel, and the training goal includes the difference between the image rendered by the Gaussian kernel corresponding to each point in the dense point cloud and the image of the target area; after the training is completed or during the training process, deleting some Gaussian kernels according to the importance of each Gaussian kernel.

[0094] As one of the feasible ways, the Gaussian splash unit 503 can be configured as follows: in the first N rounds of training iterations, after frequency domain analysis and high-frequency noise filtering of the loss function, the parameters of the Gaussian kernel are iteratively updated using the loss function after high-frequency noise filtering, where the loss function is determined based on the training objective, and N is less than or equal to the total number of training rounds.

[0095] As one of the achievable methods, the Gaussian splash unit 503 may be configured to: when rendering the Gaussian kernel to obtain an image, perform an affine transformation on the color of the pixel points in the rendered image according to an affine matrix, wherein the affine matrix is ​​updated during the training process.

[0096] As one of the feasible ways, the opacity representation is obtained based on the opacity and the mask value; the training goal also includes: maximizing the number of Gaussian kernels whose mask values ​​meet the preset conditions; after the training, when the Gaussian splash unit 503 deletes some Gaussian kernels according to the importance of each Gaussian kernel, it can be configured to: delete the Gaussian kernels whose mask values ​​meet the preset conditions based on the mask values ​​of each Gaussian kernel obtained through training.

[0097] As one of the feasible ways, when the Gaussian splattering unit 503 deletes some Gaussian kernels according to their importance during the training process, it can be configured as follows: in each round of training iteration, some Gaussian kernels are deleted according to their importance so that the number of Gaussian kernels retained after the training ends meets the preset quantity requirement; wherein the importance of the Gaussian kernel is determined based on the opacity of the Gaussian kernel, or based on the opacity of the Gaussian kernel and the size of the Gaussian kernel, and the scale of the Gaussian kernel is related to the standard deviation of the Gaussian distribution.

[0098] As one of the feasible ways, when the point cloud generation unit 502 generates a dense point cloud using target area images from multiple perspectives, it can be configured as follows: performing motion recovery structure processing on the target area images from multiple perspectives to obtain camera parameters and posture information corresponding to each target image; performing multi-perspective stereo vision estimation using the camera parameters and posture information to obtain depth maps corresponding to the multiple perspectives; filtering the depth information of the depth maps corresponding to the multiple perspectives based on filtering parameters, wherein the filtering parameters are determined based on the depth information estimation status corresponding to each depth map; and fusing the filtered depth maps to obtain a dense point cloud.

[0099] As one of the feasible ways, the screening parameters include a confidence threshold. When the point cloud generation unit 502 screens the depth information of the depth maps corresponding to multiple perspectives according to the screening parameters, it can be configured as follows: determining the confidence threshold of each depth map according to the depth confidence distribution corresponding to each depth map; and filtering out the depth information in each depth map whose confidence is less than or equal to the corresponding confidence threshold.

[0100] As one of the feasible ways, when the point cloud generation unit 502 determines the confidence threshold of each depth map according to the depth confidence distribution corresponding to each depth map, it can be configured as follows: determining the confidence threshold of the depth map according to the depth confidence corresponding to the probability peak in the probability distribution of the depth confidence of the depth map.

[0101] As one of the feasible methods, the screening parameters include a distance error threshold. When the point cloud generation unit 502 screens the depth information of the depth maps corresponding to multiple perspectives according to the screening parameters, it can be configured as follows: based on the shooting distance of the target area image to which each depth map belongs, respectively determine the distance error threshold of each depth map; for the depth information of pixel point i on the depth map corresponding to perspective a, if the distance between the depth information of pixel point i obtained by mapping the depth maps corresponding to other perspectives to perspective a exceeds the distance error threshold of the depth map corresponding to perspective a, then filter out the depth information of pixel point i on the depth map corresponding to perspective a, where perspective a is any perspective among the multiple perspectives, and pixel point i is any point on the target area image corresponding to perspective a.

[0102] As one of the feasible methods, the screening parameters include the number of observation points. When the point cloud generation unit 502 screens the depth information of the depth maps corresponding to multiple perspectives according to the screening parameters, it can be configured as follows: spatially aligning the depth maps corresponding to the multiple perspectives, taking each spatial point as a target point, and determining the number of observation points corresponding to the target point according to the depth confidence distribution of the multiple observation points corresponding to the target point, the observation point being the point in the depth maps corresponding to the multiple perspectives that is aligned with the target point; for multiple observation points aligned with the same target point, screening the number of observation points corresponding to the target point from the multiple observation points.

[0103] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0104] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0105] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0106] And an electronic device, comprising:

[0107] one or more processors; and

[0108] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0109] The present application also provides a computer program product, including a computer program, which implements the steps of any one of the methods in the aforementioned method embodiments when executed by a processor.

[0110] in, Figure 6 The architecture of the electronic device is shown as an example, which may include a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, and a memory 620. The processor 610, the video display adapter 611, the disk drive 612, the input / output interface 613, the network interface 614, and the memory 620 may be communicatively connected via a communication bus 630.

[0111] Among them, the processor 610 can be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in this application.

[0112] The memory 620 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 620 can store an operating system 621 for controlling the operation of the electronic device 600, and a basic input and output system (BIOS) 622 for controlling the low-level operation of the electronic device 600. In addition, a web browser 623, a data storage management system 624, and a three-dimensional reconstruction device 625, etc. can also be stored. The above-mentioned point cloud generation device / three-dimensional reconstruction device 625 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided in the present application is implemented by software or firmware, the relevant program code is stored in the memory 620 and is called and executed by the processor 610.

[0113] The input / output interface 613 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0114] The network interface 614 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0115] The bus 630 comprises a pathway for transmitting information between the various components of the device (eg, the processor 610, the video display adapter 611, the disk drive 612, the input / output interface 613, the network interface 614, and the memory 620).

[0116] It should be noted that, although the above device only shows a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, a memory 620, a bus 630, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0117] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0118] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A three-dimensional reconstruction method, characterized in that: The method comprises: Acquire target area images from multiple perspectives; Generate a dense point cloud using the target area images from the multiple perspectives; The dense point cloud is subjected to Gaussian splash processing to obtain a three-dimensional model of the target area; wherein, in the Gaussian splash processing, some Gaussian kernels are deleted according to the importance of each Gaussian kernel, and the importance of the Gaussian kernel is determined at least based on the opacity of the Gaussian kernel.

2. The method according to claim 1, characterized in that Performing Gaussian splatting processing on the dense point cloud to obtain a three-dimensional model of the target area includes: Converting each point in the dense point cloud into a Gaussian kernel, wherein parameters of the Gaussian kernel include Gaussian distribution representation, color representation, and opacity representation corresponding to the point in the dense point cloud; Training the parameters of the Gaussian kernel, wherein the training objective includes the difference between an image rendered by the Gaussian kernel corresponding to each point in the dense point cloud and the image of the target area; After the training is completed or during the training, some Gaussian kernels are deleted according to the importance of each Gaussian kernel.

3. The method according to claim 2, characterized in that: During the first N rounds of iterations of the training, after frequency domain analysis and high-frequency noise filtering of the loss function, the parameters of the Gaussian kernel are iteratively updated using the loss function after high-frequency noise filtering. The loss function is determined based on the goal of the training, and N is less than or equal to the total number of rounds of the training.

4. The method according to claim 2, characterized in that: When the Gaussian kernel is rendered to obtain an image, the colors of the pixels in the rendered image are affine transformed according to an affine matrix, wherein the affine matrix is ​​updated during the training process.

5. The method according to claim 2, characterized in that: The opacity representation is obtained based on the opacity and mask values; The training objectives also include: maximizing the number of Gaussian kernels whose mask values ​​meet preset conditions; After the training is completed, deleting some Gaussian kernels according to the importance of each Gaussian kernel includes: according to the mask value of each Gaussian kernel obtained through training, deleting the Gaussian kernel whose mask value meets the preset condition.

6. The method according to claim 2, characterized in that In the training process, deleting some Gaussian kernels according to the importance of each Gaussian kernel includes: In each iteration of the training, some Gaussian kernels are deleted according to the importance of each Gaussian kernel so that the number of Gaussian kernels retained after the training is completed meets the preset quantity requirement; The importance of the Gaussian kernel is determined based on the opacity of the Gaussian kernel, or based on the opacity of the Gaussian kernel and the size of the Gaussian kernel, and the size of the Gaussian kernel is related to the standard deviation of the Gaussian distribution.

7. The method according to any one of claims 1 to 6, characterized in that Using the target area images from the multiple perspectives, generating a dense point cloud includes: Performing structure-from-motion processing on the target area images of the multiple perspectives to obtain camera parameters and posture information corresponding to each target image; Using the camera parameters and the pose information to perform multi-view stereo vision estimation to obtain depth maps corresponding to multiple viewpoints; Filtering depth information of the depth maps corresponding to the multiple perspectives according to a filtering parameter, wherein the filtering parameter is determined according to an estimation status of the depth information corresponding to each depth map; The filtered depth maps are fused to obtain a dense point cloud.

8. The method according to claim 7, characterized in that The screening parameter includes a confidence threshold, and screening the depth information of the depth maps corresponding to the multiple perspectives according to the screening parameter includes: Determine the confidence threshold of each depth map according to the depth confidence distribution corresponding to each depth map; Filter out the depth information in each depth map whose confidence is less than or equal to the corresponding confidence threshold.

9. The method according to claim 7, characterized in that: The screening parameter includes the number of observation points, and screening the depth information of the depth maps corresponding to the multiple perspectives according to the screening parameter includes: Spatially aligning the depth maps corresponding to the multiple perspectives, taking each spatial point as a target point, and determining the number of observation points corresponding to the target point according to the depth confidence distribution of multiple observation points corresponding to the target point, wherein the observation point is a point in the depth maps corresponding to the multiple perspectives that is aligned with the target point; For multiple observation points aligned with the same target point, the number of observation points corresponding to the target point is selected from the multiple observation points.

10. A three-dimensional reconstruction device, characterized in that: The three-dimensional reconstruction device comprises: An image acquisition unit, configured to acquire target area images from multiple viewing angles; A point cloud generating unit is configured to generate a dense point cloud using the target area images of the multiple perspectives; A Gaussian splash unit is configured to perform Gaussian splash processing on the dense point cloud to obtain a three-dimensional model of the target area; wherein, in the Gaussian splash processing, some Gaussian kernels are deleted according to the importance of each Gaussian kernel, and the importance of the Gaussian kernel is determined at least based on the opacity of the Gaussian kernel.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

12. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of claims 1 to 9.

Images