Three-dimensional scene reconstruction method and device, electronic equipment and storage medium
By constructing an initial 3D scene model and using cloth simulation filtering and cluster analysis to remove vehicle point clouds, the problem of vehicle noise interference in 3D modeling was solved, generating a high-quality 3D scene model and improving construction efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-11
- Publication Date
- 2026-03-31
AI Technical Summary
In existing technologies, the point cloud data of moving vehicles during the 3D modeling process causes floating and misalignment when matching noisy point clouds, affecting the model quality. Furthermore, manually identifying vehicle point clouds is time-consuming and laborious, making it difficult to scale up applications.
An initial 3D scene model is constructed by acquiring a dataset of oblique images. This model is then converted into raw 3D point cloud data for filtering. A cloth simulation filtering algorithm and cluster analysis are used to identify and remove vehicle point clouds. A triangulation algorithm is then used to repair the void areas, generating a 3D scene model without vehicles.
It achieves accurate extraction and removal of vehicle point cloud data, improves the spatial consistency and visual realism of 3D models, solves the problems of 3D reconstruction distortion and texture disorder, and improves construction efficiency.
Smart Images
Figure CN121767589A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of 3D modeling technology, and more specifically, to a 3D scene reconstruction method, apparatus, electronic device, and storage medium. Background Technology
[0002] With the rapid development of 3D perception technology, 3D modeling has been widely applied in fields such as autonomous driving, high-precision map construction, robot navigation, and urban digital twins. These applications typically require the acquisition of 3D point cloud data of the environment based on LiDAR, depth cameras, or multi-view vision systems, and the generation of accurate 3D scene models through fusion and reconstruction algorithms. However, in the process of real-world 3D modeling, image data containing a large number of moving vehicles is often captured, and the positions of these vehicles change continuously at different time stamps. If their point cloud data is directly used in the 3D model construction process, it leads to a large number of floating and misaligned noisy point clouds during point cloud matching, ultimately forming visual anomalies such as "streaking" or "ghosting" during the texture mapping stage, severely affecting model quality. Therefore, in the construction of 3D tilted models, it is necessary to remove moving vehicles from the 3D scene model to improve the quality of the 3D model.
[0003] Currently, the main method relies on professionals to manually identify and delete vehicle-related point clouds or meshes. This is not only time-consuming and labor-intensive, but also depends on subjective judgment, making it difficult to scale up and hindering the large-scale and rapid production of high-precision real-scene 3D data. Summary of the Invention
[0004] The purpose of this application is to provide a three-dimensional scene reconstruction method, apparatus, electronic device and storage medium to address the shortcomings of the prior art, so as to solve the technical problems existing in the prior art.
[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, embodiments of this application provide a three-dimensional scene reconstruction method, the method comprising: Obtain an oblique image dataset, which includes image data from multiple viewpoints; Based on the oblique image dataset, an initial 3D scene model is constructed, which includes vehicles. The initial 3D scene model is converted into raw 3D point cloud data, and the raw 3D point cloud data is filtered to obtain ground point cloud set and non-ground point cloud data. Vehicle point cloud data is identified from the non-ground point cloud data, and the vehicle point cloud data is removed from the original 3D point cloud data to obtain point cloud data after removing vehicles. Based on the point cloud data after removing vehicles, 3D reconstruction is performed to generate a 3D scene model, which does not contain vehicles.
[0006] Optionally, filtering the original 3D point cloud data to obtain ground point cloud data and non-ground point data includes: The original 3D point cloud data is flipped along the Z-axis to obtain the flipped 3D point cloud data. The ground in the flipped 3D point cloud data is located on the upper layer of the non-ground layer. A cloth simulation filtering algorithm is used to extract ground point cloud data from the flipped 3D point cloud data, resulting in non-ground point data that are not covered by the cloth.
[0007] Optionally, the step of employing a cloth simulation filtering algorithm to extract ground point cloud data from the flipped 3D point cloud data to obtain non-ground point data not covered by the cloth includes: Set the initial height of the fabric, the grid resolution, and the gravitational acceleration; Simulate the falling of the fabric. During the simulation of the falling of the fabric, when a node in the fabric touches any point cloud in the flipped 3D point cloud data, control the node to stop falling and transmit support force to the neighboring node. If the support force is greater than or equal to the gravity, control the neighboring fabric node to stop moving. Iterate until all nodes in the fabric reach a stationary state, forming a fabric coverage area that fits the ground. The point cloud area within the fabric coverage area is used as ground point cloud data, and the point cloud area outside the fabric coverage area is used as non-ground point data.
[0008] Optionally, identifying vehicle point cloud data from the non-terrestrial point cloud data includes: Cluster analysis is performed on the non-ground point cloud data to obtain the vehicle point cloud data.
[0009] Optionally, the step of performing cluster analysis on the non-ground point cloud data to obtain the vehicle point cloud data includes: A pre-defined clustering algorithm is used to cluster the non-ground point cloud data to obtain multiple point cloud clusters; Determine the directed bounding box corresponding to each point cloud cluster; The directed bounding boxes corresponding to each point cloud cluster are compared with preset vehicle geometric features, and the point cloud clusters corresponding to the directed bounding boxes that satisfy the vehicle geometric features are taken as the vehicle point cloud data.
[0010] Optionally, the step of performing 3D reconstruction based on the point cloud data after removing vehicles to generate a 3D scene model includes: Based on the point cloud data after removing the vehicles, the hollow areas formed in the initial 3D scene model are determined; A pre-defined triangulation algorithm is used to reconstruct the mesh in the hollow region, generating new triangular patches. Based on the tilted image dataset, a texture map corresponding to the triangular facet is generated, and the texture map corresponding to the triangular facet is mapped to the hole region to obtain the repaired 3D real-world point cloud data. The repaired 3D real-world point cloud data is then flipped to obtain the 3D scene model.
[0011] Optionally, generating the texture map corresponding to the triangular facet based on the tilted image dataset includes: Determine the visibility of each vertex on the triangular facet in the tilted image dataset; Based on the visibility of the vertex in the oblique image dataset, oblique image data from the target viewpoint is selected, and the color information of the vertex is extracted from the oblique image data from the target viewpoint. Based on the color information corresponding to all vertices on the triangular facet, a texture map corresponding to the triangular facet is generated.
[0012] Secondly, embodiments of this application also provide a three-dimensional scene reconstruction apparatus, the apparatus comprising: The acquisition module is used to acquire an oblique image dataset, which includes image data from multiple viewpoints. A construction module is used to construct an initial three-dimensional scene model based on the oblique image dataset, wherein the initial three-dimensional scene model includes vehicles; The conversion module is used to convert the initial 3D scene model into raw 3D point cloud data; The filtering module is used to filter the original 3D point cloud data to obtain ground point cloud set and non-ground point cloud data; The elimination module is used to identify vehicle point cloud data from the non-ground point cloud data and eliminate the vehicle point cloud data from the original three-dimensional point cloud data to obtain point cloud data after vehicle elimination. The construction module is also used to perform three-dimensional reconstruction based on the point cloud data after removing vehicles, and generate a three-dimensional scene model, which does not contain vehicles.
[0013] Optionally, the filtering module is specifically used for: The original 3D point cloud data is flipped along the Z-axis to obtain the flipped 3D point cloud data. The ground in the flipped 3D point cloud data is located on the upper layer of the non-ground layer. A cloth simulation filtering algorithm is used to extract ground point cloud data from the flipped 3D point cloud data, resulting in non-ground point data that are not covered by the cloth.
[0014] Optionally, the filtering module is specifically used for: Set the initial height of the fabric, the grid resolution, and the gravitational acceleration; Simulate the falling of the fabric. During the simulation of the falling of the fabric, when a node in the fabric touches any point cloud in the flipped 3D point cloud data, control the node to stop falling and transmit support force to the neighboring node. If the support force is greater than or equal to the gravity, control the neighboring fabric node to stop moving. Iterate until all nodes in the fabric reach a stationary state, forming a fabric coverage area that fits the ground. The point cloud area within the fabric coverage area is used as ground point cloud data, and the point cloud area outside the fabric coverage area is used as non-ground point data.
[0015] Optionally, the rejection module is specifically used for: Cluster analysis is performed on the non-ground point cloud data to obtain the vehicle point cloud data.
[0016] Optionally, the rejection module is specifically used for: A pre-defined clustering algorithm is used to cluster the non-ground point cloud data to obtain multiple point cloud clusters; Determine the directed bounding box corresponding to each point cloud cluster; The directed bounding boxes corresponding to each point cloud cluster are compared with preset vehicle geometric features, and the point cloud clusters corresponding to the directed bounding boxes that satisfy the vehicle geometric features are taken as the vehicle point cloud data.
[0017] Optionally, the building module is further configured to: Based on the point cloud data after removing the vehicles, the hollow areas formed in the initial 3D scene model are determined; A pre-defined triangulation algorithm is used to reconstruct the mesh in the hollow region, generating new triangular patches. Based on the tilted image dataset, a texture map corresponding to the triangular facet is generated, and the texture map corresponding to the triangular facet is mapped to the hole region to obtain the repaired 3D real-world point cloud data. The repaired 3D real-world point cloud data is then flipped to obtain the 3D scene model.
[0018] Optionally, the building module is specifically used for: Determine the visibility of each vertex on the triangular facet in the tilted image dataset; Based on the visibility of the vertex in the oblique image dataset, oblique image data from the target viewpoint is selected, and the color information of the vertex is extracted from the oblique image data from the target viewpoint. Based on the color information corresponding to all vertices on the triangular facet, a texture map corresponding to the triangular facet is generated.
[0019] Thirdly, embodiments of this application provide an electronic device, which includes a memory for storing one or more programs and a processor. When the one or more programs are executed by the processor, the above-described three-dimensional scene reconstruction method is implemented.
[0020] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described three-dimensional scene reconstruction method.
[0021] The beneficial effects of this application are: This application provides a method, apparatus, electronic device, and storage medium for three-dimensional scene reconstruction. First, an initial three-dimensional scene model is constructed using the oblique image dataset. This initial three-dimensional scene model includes vehicles. Then, the initial three-dimensional scene model is converted into raw three-dimensional point cloud data. Ground point filtering is performed on the raw three-dimensional point cloud data to obtain non-ground point cloud data. Vehicle point cloud data corresponding to all vehicles included in the initial three-dimensional scene model is identified from the non-ground point cloud data. This vehicle point cloud data is then removed from the raw three-dimensional point cloud data, resulting in point cloud data after vehicle removal. This achieves accurate extraction and removal of vehicle point cloud data. Finally, three-dimensional reconstruction is performed based on the point cloud data after vehicle removal to generate a three-dimensional scene model free from vehicle interference. This ensures the spatial consistency and visual realism of the final three-dimensional scene model, solving problems such as three-dimensional reconstruction distortion, texture misalignment, and structural distortion caused by the presence of road vehicles in existing technologies. Furthermore, the entire reconstruction process requires no manual intervention, improving the efficiency of three-dimensional model construction.
[0022] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0024] Figure 1 A flowchart illustrating a three-dimensional scene reconstruction method provided in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the acquisition of tilted image data in a three-dimensional scene reconstruction method provided in an embodiment of this application; Figure 3 A flowchart illustrating the construction of a 3D model provided in this application embodiment; Figure 4 This is a schematic diagram of point cloud matching provided for an embodiment of this application; Figure 5 This is a schematic diagram of TIN network construction provided in an embodiment of this application; Figure 6 This is a schematic diagram of the white model construction provided in an embodiment of this application; Figure 7 The renderings of the real-world 3D model provided in the embodiments of this application; Figure 8 A schematic diagram illustrating the display of a three-dimensional model of a latte art provided in an embodiment of this application; Figure 9 A flowchart illustrating another three-dimensional scene reconstruction method provided in this application embodiment; Figure 10 A schematic diagram illustrating the simulated fall of fabric provided in an embodiment of this application; Figure 11 A flowchart illustrating another three-dimensional scene reconstruction method provided in this application embodiment; Figure 12 A flowchart illustrating another three-dimensional scene reconstruction method provided in this application embodiment; Figure 13 An example of a directed bounding box provided in this application embodiment; Figure 14 This is a schematic diagram of the vehicle point cloud corresponding to the vehicle model provided in the embodiments of this application; Figure 15 A flowchart illustrating another three-dimensional scene reconstruction method provided in this application embodiment; Figure 16 A flowchart illustrating another three-dimensional scene reconstruction method provided in this application embodiment; Figure 17 This is a schematic diagram of the structure of a three-dimensional scene reconstruction device provided in an embodiment of this application; Figure 18 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0026] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.
[0027] The following will explain the implementation principle and corresponding beneficial effects of the three-dimensional scene reconstruction method provided in this application through several specific embodiments.
[0028] Optionally, refer to Figure 1 The diagram shown is a flowchart illustrating a three-dimensional scene reconstruction method provided in an embodiment of this application; optionally, the execution subject of this method can be an electronic device with data processing capabilities, such as a computer or server. Figure 1 As shown, the method includes: S101. Obtain the oblique image dataset.
[0029] The oblique image dataset includes image data from multiple perspectives. For example, the input sources of the image data may include: oblique photogrammetric images (acquired by a five-lens drone), LiDAR point cloud data, and high-precision orthophotos (DOM).
[0030] In one feasible approach, as referenced Figure 2 As shown, after determining the area of the 3D scene to be constructed, a drone system or a large aircraft equipped with multiple cameras is used to photograph the target area from multiple angles, such as vertical and tilted, to obtain image data from multiple perspectives.
[0031] Optionally, after acquiring image data from multiple perspectives, preprocessing of the image data is also required. Preprocessing operations include noise removal, distortion correction, and motion compensation, as detailed below: (1) Noise Removal: When acquiring image data, some noise points will inevitably appear in the data due to the influence of equipment accuracy, operator experience, environmental factors, etc. Therefore, image processing algorithms, such as median filtering and bilateral filtering, are used to denoise the images, eliminate sensor noise and atmospheric disturbance, and improve image clarity and detail; (2) Distortion correction: The radial and tangential distortion of the image is corrected by using the camera calibration parameters (intrinsic parameter matrix and distortion coefficients) so that the shape and size of the objects in the image are consistent with the actual shape and size. (3) Motion compensation: Based on the acquired data, the flight trajectory and attitude information of the aircraft are used to perform motion compensation on the images to eliminate the blur caused by flight and improve the image clarity.
[0032] S102. Construct an initial 3D scene model based on the oblique image dataset.
[0033] The initial 3D scene model includes vehicles.
[0034] In one feasible approach, a 3D model is constructed using the acquired oblique image dataset, the specific process of which is as follows: Figure 3 As shown: (1) Point cloud matching: If SIFT or SuperPoint algorithms are used to detect key points, point cloud matching calculations are performed on all stereo image pairs contained in the block to automatically generate high-density point cloud data, with the following effect: Figure 4 As shown.
[0035] (2) TIN network construction: High-density point cloud data is thinned and processed with normal vectors, and an irregular triangular network (TIN) is automatically constructed, with the following effect: Figure 5 As shown.
[0036] (3) White model generation: A white model is generated using a TIN mesh, with the following effect: Figure 6 As shown.
[0037] (4) Texture mapping: Based on spatial relationships, the textures corresponding to the white model are automatically acquired from the image and mapped, and the white model is given realistic textures to construct a complete initial 3D scene model, as shown in the image. Figure 7 As shown.
[0038] The initial 3D scene model includes all static and dynamic objects in the real scene, such as buildings, roads, greenery, and parked or moving vehicles.
[0039] Optionally, such as Figure 8The image shown is a schematic diagram of the pattern display of another initial 3D scene model. Therefore, it is necessary to remove the vehicle model from the initial 3D scene model to obtain a high-precision 3D real scene model without vehicle interference, so as to solve the problems of 3D reconstruction distortion, texture disorder and structural distortion caused by the presence of road vehicles in the existing technology.
[0040] S103. Convert the initial 3D scene model into raw 3D point cloud data, and filter the raw 3D point cloud data to obtain ground point cloud set and non-ground point cloud data.
[0041] Optionally, in order to accurately identify non-ground objects (such as vehicles), it is necessary to effectively separate ground points from non-ground points. Therefore, in this embodiment, it is proposed that the initial 3D scene model be converted into raw 3D point cloud data first, and then the raw 3D point cloud data be filtered to divide the raw 3D point cloud data into two categories: ground point cloud set and non-ground point cloud data, thereby achieving the separation of ground points from non-ground points.
[0042] Ground point cloud data corresponds to ground surfaces such as roads and sidewalks; non-ground point cloud data corresponds to objects above ground such as buildings, trees, streetlights, vehicles, and pedestrians.
[0043] S104. Identify vehicle point cloud data from non-ground point cloud data, remove vehicle point cloud data from the original 3D point cloud data to obtain point cloud data after removing vehicles, and perform 3D reconstruction based on the point cloud data after removing vehicles to generate a 3D scene model.
[0044] The 3D scene model does not include vehicles.
[0045] In one feasible approach, a segmentation algorithm can be used to process non-terrestrial point clouds to identify which point cloud data belong to "vehicles." These "vehicle point cloud data" are then removed from the original 3D point cloud data, forming "point cloud data after vehicle removal," meaning the point cloud data after vehicle removal does not contain vehicles. A 3D reconstruction algorithm, such as Delaunay triangulation, is then used to reconstruct the 3D scene model from the "point cloud data after vehicle removal." The final output 3D model retains static elements such as buildings, terrain, and vegetation, but completely excludes any vehicles. This achieves precise removal of dynamic vehicles in 3D space without affecting other ground features, effectively eliminating 3D reconstruction artifacts caused by vehicle presence, improving the spatial consistency and visual realism of the final 3D scene model. Furthermore, the entire reconstruction process requires no manual intervention, improving the efficiency of 3D model construction.
[0046] In summary, this application provides a method for three-dimensional scene reconstruction. First, an initial three-dimensional scene model is constructed using the oblique image dataset, which includes vehicles. Then, the initial three-dimensional scene model is converted into raw three-dimensional point cloud data, and ground point filtering is applied to the raw three-dimensional point cloud data to obtain non-ground point cloud data. Vehicle point cloud data corresponding to all vehicles included in the initial three-dimensional scene model is identified from the non-ground point cloud data and removed from the raw three-dimensional point cloud data, resulting in point cloud data after vehicle removal. This achieves accurate extraction and removal of vehicle point cloud data. Finally, three-dimensional reconstruction is performed based on the point cloud data after vehicle removal to generate a three-dimensional scene model free from vehicle interference. This ensures the spatial consistency and visual realism of the final three-dimensional scene model, solving problems such as three-dimensional reconstruction distortion, texture misalignment, and structural distortion caused by the presence of road vehicles in existing technologies. Furthermore, the entire reconstruction process requires no manual intervention, improving the efficiency of three-dimensional model construction.
[0047] Optionally, refer to Figure 9 As shown, in step S103 above, the original 3D point cloud data is filtered to obtain ground point cloud data and non-ground point data, including: S201. Flip the original 3D point cloud data along the Z-axis to obtain the flipped 3D point cloud data.
[0048] In the flipped 3D point cloud data, the ground is located on the upper layer of the non-ground layer.
[0049] S202. Using a cloth simulation filtering algorithm, ground point cloud is extracted from the flipped 3D point cloud data to obtain non-ground point data not covered by the cloth.
[0050] Among them, the Cloth Simulation Filtering (CSF) algorithm utilizes the physical phenomenon that gravity causes objects to adhere to the Earth's surface to perform ground point filtering.
[0051] In one feasible approach, as referenced Figure 10 As shown, assuming the ground isn't excessively steep, the original 3D point cloud data is flipped along the Z-axis to obtain the flipped 3D point cloud data. After the flip, ground points (low elevation) that were originally below become higher points, while non-ground points such as buildings and vegetation that were originally above are moved below. Therefore, in the new spatial layout, the real ground is now visually at the "top," while non-ground objects (such as tree canopies and rooftops) are sunken below. Then, a soft and somewhat resilient fabric is simulated to cover the flipped ground, as shown... Figure 10As shown by the dashed line, this imaginary cloth adheres to the flipped ground due to gravity. The part of the imaginary cloth that adheres to the ground is the ground. Data on non-ground points not covered by the cloth is obtained, realizing the separation of ground points and non-ground points and effectively reducing the problem of missing ground points.
[0052] Optionally, refer to Figure 11 As shown, step S202 above includes: S301, Set the initial height of the fabric, the grid resolution, and the gravitational acceleration.
[0053] S302. Simulate the falling of fabric. During the simulation of the falling fabric, when a node in the fabric touches any point cloud in the flipped 3D point cloud data, control the node to stop falling and transfer support force to the neighboring node. If the support force is greater than or equal to the gravity, control the neighboring fabric nodes to stop moving. Iterate until all nodes in the fabric reach a stationary state, forming a fabric coverage area that fits the ground.
[0054] S303. The point cloud area within the fabric coverage area is used as ground point cloud data, and the point cloud area outside the fabric coverage area is used as non-ground point data.
[0055] In one feasible approach, before simulating the cloth's fall, cloth initialization is required. This involves setting the cloth as a two-dimensional mesh structure with a resolution matching the point cloud density; setting the initial height to a certain distance above the highest point (e.g., +5 meters); and defining the unit gravitational acceleration g = 9.8 m / s². Then, the cloth's fall is simulated. During the fall simulation, each node in the cloth continuously moves downwards under the influence of gravity. When a node touches any point cloud in the flipped 3D point cloud data, the node stops falling and transmits support force to surrounding nodes. If the support force is insufficient to resist gravity, the fall continues. If the support force is greater than or equal to gravity, i.e., sufficient to resist gravity, adjacent cloth nodes stop moving. This process is iterated until all nodes in the cloth reach a stationary state, forming a cloth-covered area that conforms to the ground. The point cloud area ultimately covered by the cloth is considered the "ground point," and the remaining uncovered points are considered "non-ground points," thus achieving effective separation between ground and non-ground points. Compared with traditional slope filtering or elevation difference methods, the cloth simulation algorithm used in this application can better adapt to complex terrain (such as steps and gentle slopes), avoid accidentally deleting low buildings or objects under overpasses, effectively reduce the omission of ground points, and improve the completeness and robustness of ground point extraction.
[0056] Optionally, identifying vehicle point cloud data from non-terrestrial point cloud data in step S104 above includes: Cluster analysis was performed on the non-terrestrial point cloud data to obtain vehicle point cloud data.
[0057] In one feasible approach, pre-defined clustering algorithms, such as Euclidean clustering, can be used to perform clustering analysis on non-terrestrial point cloud data to obtain vehicle point cloud data.
[0058] Optionally, refer to Figure 12 As shown, the above clustering analysis of non-terrestrial point cloud data yields vehicle point cloud data, including: S401. Using a pre-defined clustering algorithm, cluster the non-ground point cloud data to obtain multiple point cloud clusters.
[0059] S402. Determine the directed bounding box corresponding to each point cloud cluster.
[0060] It's important to note that a directed bounding box is not simply drawing a rectangle; its key feature is that this rectangle can be rotated to more closely fit the outline of the target object, as shown below. Figure 13 As shown.
[0061] S403. Compare the directed bounding boxes corresponding to each point cloud cluster with the preset vehicle geometric features, and take the point cloud clusters corresponding to the directed bounding boxes that satisfy the vehicle geometric features as vehicle point cloud data.
[0062] In one feasible approach, such as using Euclidean clustering, the non-terrestrial point cloud data can be clustered to obtain multiple point cloud clusters C1, C2, ..., Cn, each cluster corresponding to an object. Then, the dimensions (length, width, and height) of the directed bounding box for each point cloud cluster are calculated, and these bounding boxes are compared with preset vehicle geometric features. For example, if the vehicle geometric features are: length ≈ 4–6m, width ≈ 1.8–2.5m, and height ≈ 1.5–2m, and the directed bounding box corresponding to point cloud cluster C1 satisfies the vehicle geometric features, then point cloud cluster C1 can be identified as a vehicle. (See reference...) Figure 14 As shown, the vehicle point cloud data of all vehicle models were obtained using the above method.
[0063] Optionally, refer to Figure 15 As shown, step S104 above, which involves performing 3D reconstruction based on the point cloud data after removing vehicles to generate a 3D scene model, includes: S501. Based on the point cloud data after removing vehicles, determine the void regions formed in the initial 3D scene model.
[0064] Among them, the void region refers to the area on the surface of the 3D model that should exist but is currently not covered by effective point cloud.
[0065] It should be noted that "point cloud data after removing vehicles" refers to the point cloud data remaining after removing vehicle point cloud data from the original 3D point cloud data. Because some structures (such as vehicles) have been removed, the spaces originally occupied by these objects are now empty—forming "holes"—leading to significant inconsistencies in the triangular faces of the reconstructed model. Therefore, boundary extraction is performed on the point cloud data after removing vehicles, that is, identifying the vertex sequences on the edges of the holes to form closed contour lines, in order to obtain at least one hole region formed in the initial 3D scene model.
[0066] S502. Using a pre-defined triangulation algorithm, the hollow area is reconstructed into a mesh, generating new triangular patches.
[0067] In one feasible approach, the Delaunay triangulation algorithm is used to reconstruct the mesh for the void region, as follows: (1) Construct a “super triangle” containing all boundary points and put it into the triangle linked list; (2) Insert the scattered points in the point set in sequence, find the triangle whose circumcircle contains the insertion point in the triangle linked list (called the influence triangle of the point), delete the common edge of the influence triangle, connect the insertion point with all the vertices of the influence triangle, and complete the insertion of a point in the Delaunay triangle linked list. (3) Optimize the newly formed triangles locally according to the optimization criteria, and put the newly formed triangles into the Delaunay triangle linked list.
[0068] (4) Repeat step 2 above until all scattered points are inserted, generating new triangular patches, thus achieving reasonable filling of the holes.
[0069] S503. Based on the oblique image dataset, generate texture maps corresponding to triangular patches, and map the texture maps corresponding to triangular patches to the hole areas to obtain repaired 3D real-world point cloud data. Then, flip the repaired 3D real-world point cloud data to obtain a 3D scene model.
[0070] Optionally, after the hole is repaired, a texture needs to be applied to the new triangular facet to ensure that the texture of the repaired area blends seamlessly with the surrounding road, achieving visual fusion. In this embodiment, a multi-view texture mapping algorithm is used to generate texture maps corresponding to the triangular facets, and these texture maps are mapped to the hole area to obtain the repaired 3D real-world point cloud data. The repaired 3D real-world point cloud data is then flipped to obtain a 3D scene model. This effectively removes dynamic vehicle interference during the 3D reconstruction of tilted images, thereby generating a high-fidelity 3D real-world model without dynamic objects, ensuring that the generated 3D scene model reflects a realistic and high-quality real-world scene.
[0071] Optionally, refer to Figure 16 As shown, in step S503 above, generating the texture map corresponding to the triangular facet based on the tilted image dataset includes: S601. Determine the visibility of each vertex on the triangular facet in the tilted image dataset.
[0072] S602. Based on the visibility of vertices in the oblique image dataset, select oblique image data from the target viewpoint and extract the color information of vertices from the oblique image data from the target viewpoint.
[0073] S603. Generate a texture map corresponding to the triangle based on the color information of all vertices on the triangle.
[0074] In one feasible approach, the repaired 3D mesh is determined. For each point on the new mesh, its visibility across all image data is calculated. From the image data that shows the real road surface and ground at that point, images that are occluded, too far away, or at an excessively skewed angle are excluded. The optimal viewing angle (such as the highest resolution and the most positive angle) is selected to extract color information. Since the resolution of the texture image is limited, various texture filtering techniques may be needed to solve problems such as jagged edges and blurring, thereby improving the final rendering quality and effect. The color information of all points is then fused to generate a new, realistic texture map. At this point, the texture map corresponding to the triangular facet is generated.
[0075] Based on the same inventive concept, this application also provides a three-dimensional scene reconstruction device corresponding to the three-dimensional scene reconstruction method. Since the principle of the device in this application is similar to the three-dimensional scene reconstruction method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.
[0076] Optionally, refer to Figure 17 The diagram shown is a structural schematic of a three-dimensional scene reconstruction device provided in this application. The device includes: The acquisition module 1701 is used to acquire the oblique image dataset, which includes image data from multiple perspectives. Module 1702 is used to construct an initial 3D scene model based on the oblique image dataset. The initial 3D scene model includes vehicles. Conversion module 1703 is used to convert the initial 3D scene model into raw 3D point cloud data; Filtering module 1704 is used to filter the original 3D point cloud data to obtain ground point cloud set and non-ground point cloud data; The removal module 1705 is used to identify vehicle point cloud data from non-ground point cloud data and remove vehicle point cloud data from the original three-dimensional point cloud data to obtain point cloud data after removing vehicles. The construction module 1702 is also used to perform three-dimensional reconstruction based on the point cloud data after removing vehicles, and generate a three-dimensional scene model, which does not contain vehicles.
[0077] Optionally, the filtering module 1704 is specifically used for: The original 3D point cloud data is flipped along the Z-axis to obtain the flipped 3D point cloud data. The ground in the flipped 3D point cloud data is located on the upper layer of the non-ground layer. A cloth simulation filtering algorithm is used to extract ground point cloud data from the flipped 3D point cloud data, resulting in non-ground point data that are not covered by the cloth.
[0078] Optionally, the filtering module 1704 is specifically used for: Set the initial height of the fabric, the grid resolution, and the gravitational acceleration; Simulate the falling of the fabric. During the simulation of the falling of the fabric, when a node in the fabric touches any point cloud in the flipped 3D point cloud data, control the node to stop falling and transmit support force to the neighboring node. If the support force is greater than or equal to the gravity, control the neighboring fabric node to stop moving. Iterate until all nodes in the fabric reach a stationary state, forming a fabric coverage area that fits the ground. The point cloud area within the fabric coverage area is used as ground point cloud data, and the point cloud area outside the fabric coverage area is used as non-ground point data.
[0079] Optionally, the rejection module 1705 is specifically used for: Cluster analysis is performed on the non-ground point cloud data to obtain the vehicle point cloud data.
[0080] Optionally, the rejection module 1705 is specifically used for: A pre-defined clustering algorithm is used to cluster the non-ground point cloud data to obtain multiple point cloud clusters; Determine the directed bounding box corresponding to each point cloud cluster; The directed bounding boxes corresponding to each point cloud cluster are compared with preset vehicle geometric features, and the point cloud clusters corresponding to the directed bounding boxes that satisfy the vehicle geometric features are taken as the vehicle point cloud data.
[0081] Optionally, the building module 1702 is further configured to: Based on the point cloud data after removing the vehicles, the hollow areas formed in the initial 3D scene model are determined; A pre-defined triangulation algorithm is used to reconstruct the mesh in the hollow region, generating new triangular patches. Based on the tilted image dataset, a texture map corresponding to the triangular facet is generated, and the texture map corresponding to the triangular facet is mapped to the hole region to obtain the repaired 3D real-world point cloud data. The repaired 3D real-world point cloud data is then flipped to obtain the 3D scene model.
[0082] Optionally, the construction module 1702 is specifically used for: Determine the visibility of each vertex on the triangular facet in the tilted image dataset; Based on the visibility of the vertex in the oblique image dataset, oblique image data from the target viewpoint is selected, and the color information of the vertex is extracted from the oblique image data from the target viewpoint. Based on the color information corresponding to all vertices on the triangular facet, a texture map corresponding to the triangular facet is generated.
[0083] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.
[0084] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).
[0085] Figure 18 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device can be integrated into a terminal device or a chip of a terminal device, and the electronic device can be a computing device with data processing functions.
[0086] The electronic device includes: processor 1801 and memory 1802.
[0087] The memory 1802 is used to store programs, and the processor 1801 calls the programs stored in the memory 1802 to execute the above method embodiments. The specific implementation and technical effects are similar, and will not be described again here.
[0088] Optionally, this application also provides a program product, such as a computer-readable storage medium, including a program that, when executed by a processor, performs the above-described method embodiments.
[0089] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0090] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.
[0091] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0092] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
[0093] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.
Claims
1. A method for reconstructing a three-dimensional scene, characterized in that, The method includes: Obtain a tilted image dataset, which includes image data from multiple viewpoints; Based on the oblique image dataset, an initial 3D scene model is constructed, which includes vehicles; The initial 3D scene model is converted into raw 3D point cloud data, and the raw 3D point cloud data is filtered to obtain ground point cloud set and non-ground point cloud data. Vehicle point cloud data is identified from the non-ground point cloud data, and the vehicle point cloud data is removed from the original 3D point cloud data to obtain point cloud data after removing vehicles. Based on the point cloud data after removing vehicles, 3D reconstruction is performed to generate a 3D scene model, which does not contain vehicles.
2. The method according to claim 1, characterized in that, The filtering of the original 3D point cloud data to obtain ground point cloud data and non-ground point data includes: The original 3D point cloud data is flipped along the Z-axis to obtain the flipped 3D point cloud data. The ground in the flipped 3D point cloud data is located on the upper layer of the non-ground layer. A cloth simulation filtering algorithm is used to extract ground point cloud data from the flipped 3D point cloud data, resulting in non-ground point data that are not covered by the cloth.
3. The method according to claim 2, characterized in that, The fabric simulation filtering algorithm is used to extract ground point cloud data from the flipped 3D point cloud data, obtaining non-ground point data not covered by the fabric, including: Set the initial height of the fabric, the grid resolution, and the gravitational acceleration; Simulate the falling of the fabric. During the simulation of the falling of the fabric, when a node in the fabric touches any point cloud in the flipped 3D point cloud data, control the node to stop falling and transmit support force to the neighboring node. If the support force is greater than or equal to the gravity, control the neighboring fabric node to stop moving. Iterate until all nodes in the fabric reach a stationary state, forming a fabric coverage area that fits the ground. The point cloud area within the fabric coverage area is used as ground point cloud data, and the point cloud area outside the fabric coverage area is used as non-ground point data.
4. The method according to claim 1, characterized in that, The step of identifying vehicle point cloud data from the non-terrestrial point cloud data includes: Cluster analysis is performed on the non-ground point cloud data to obtain the vehicle point cloud data.
5. The method according to claim 4, characterized in that, The process of performing cluster analysis on the non-ground point cloud data to obtain the vehicle point cloud data includes: A pre-defined clustering algorithm is used to cluster the non-ground point cloud data to obtain multiple point cloud clusters; Determine the directed bounding box corresponding to each point cloud cluster; The directed bounding boxes corresponding to each point cloud cluster are compared with preset vehicle geometric features, and the point cloud clusters corresponding to the directed bounding boxes that satisfy the vehicle geometric features are taken as the vehicle point cloud data.
6. The method according to claim 1, characterized in that, The step of performing 3D reconstruction based on the point cloud data after removing vehicles to generate a 3D scene model includes: Based on the point cloud data after removing the vehicles, the hollow areas formed in the initial 3D scene model are determined; A pre-defined triangulation algorithm is used to reconstruct the mesh in the hollow region, generating new triangular patches. Based on the tilted image dataset, a texture map corresponding to the triangular facet is generated, and the texture map corresponding to the triangular facet is mapped to the hole region to obtain the repaired 3D real-world point cloud data. The repaired 3D real-world point cloud data is then flipped to obtain the 3D scene model.
7. The method according to claim 6, characterized in that, The step of generating texture maps corresponding to the triangular facets based on the tilted image dataset includes: Determine the visibility of each vertex on the triangular facet in the tilted image dataset; Based on the visibility of the vertex in the oblique image dataset, oblique image data from the target viewpoint is selected, and the color information of the vertex is extracted from the oblique image data from the target viewpoint. Based on the color information corresponding to all vertices on the triangular facet, a texture map corresponding to the triangular facet is generated.
8. A three-dimensional scene reconstruction device, characterized in that, The device includes: The acquisition module is used to acquire an oblique image dataset, which includes image data from multiple viewpoints. A construction module is used to construct an initial three-dimensional scene model based on the oblique image dataset, wherein the initial three-dimensional scene model includes vehicles; The conversion module is used to convert the initial 3D scene model into raw 3D point cloud data; The filtering module is used to filter the original 3D point cloud data to obtain ground point cloud set and non-ground point cloud data; The elimination module is used to identify vehicle point cloud data from the non-ground point cloud data and eliminate the vehicle point cloud data from the original three-dimensional point cloud data to obtain point cloud data after vehicle elimination. The construction module is also used to perform three-dimensional reconstruction based on the point cloud data after removing vehicles, and generate a three-dimensional scene model, which does not contain vehicles.
9. An electronic device, characterized in that, The device is characterized by comprising: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which is executed by a processor to perform the method as described in any one of claims 1-7.