Urban scale semantic segmentation benchmark dataset construction method, device and equipment

Abstract

The application relates to a city scale semantic segmentation benchmark dataset construction method, device and equipment. A city scale semantic segmentation benchmark dataset constructed by the method is a city level outdoor large scale scene dataset, which contains various data types and rich semantic labels. The dataset includes two types of 3D data: point cloud and grid, covering an accurately annotated area of 2.85 square kilometers. It includes multiple semantic labels, covering the semantic information of urban and rural scenes, and can be used as a benchmark for 3D semantic segmentation. The dataset provides original 2D images captured by a drone, with detailed semantic labels. The images can be used for research in the fields of 3D reconstruction and high resolution aerial image semantic segmentation. The dataset is applicable to different semantic segmentation benchmarks.

Description

Technical Field

[0001] This application relates to the field of dataset construction technology, and in particular to a method, apparatus and device for constructing a city-scale semantic segmentation benchmark dataset. Background Technology

[0002] In the 3D world, large-scale scene semantic segmentation assigns a specific meaning to each object in the scene, thereby aggregating complex and diverse environmental information into several specific semantic categories to facilitate planning and research. Large-scale scene semantic segmentation helps machines accurately identify object features in large scenes and plays a key role in promoting the development of scene-level robot navigation, autonomous driving, urban planning, spatial analysis, and urban fluid simulation.

[0003] To improve the machine's ability to recognize the semantics behind segmented urban data, a large amount of real-world data is typically needed for machine learning reference. Currently, the mainstream approach for large-scale semantic segmentation is based on deep learning algorithms. With the maturity of convolutional neural networks (CNNs), deep neural networks that can directly process point clouds have become popular, achieving new breakthroughs in point cloud semantic segmentation, especially for urban-scale outdoor scenes.

[0004] However, while deep neural networks that directly process grid data have achieved good results in semantic segmentation of small-scale objects, they perform poorly in large-scale scene semantic segmentation. We believe this is because publicly available large-scale scene grid datasets with fine-grained semantic annotations are very limited. Currently, the only publicly available city-level scene grid datasets are Hessigheim 3D and SUM. However, Hessigheim 3D has a scene area of ​​only 0.13 square kilometers, and some areas are unlabeled, while SUM only contains one data type (grid) and only six semantic categories, lacking the richness of information about rural and suburban scenes, making it unsuitable for research on smart city and rural development planning. Multi-view large-scale scene semantic segmentation methods are also one of the mainstream methods for 3D semantic segmentation, but currently there are no publicly available large-scale scene 3D semantic segmentation datasets with related multi-view 2D images, which to some extent limits the development of multi-view-based algorithms. Therefore, we believe that current large-scale scene semantic segmentation datasets face the following challenges: First, the scene semantic types are limited, not including various scenes from urban, suburban, and rural areas. Second, these datasets only use a single format, and the limitation of a single data type restricts the breadth of application of the datasets. Summary of the Invention

[0005] Therefore, it is necessary to address the aforementioned technical issues by providing a comprehensive benchmark dataset that can satisfy the testing of two 3D data semantic segmentation methods, namely point cloud and mesh, as well as a method, apparatus, and equipment for constructing a corresponding city-scale semantic segmentation benchmark dataset.

[0006] A method for constructing a benchmark dataset for urban-scale semantic segmentation, the method comprising:

[0007] High-resolution aerial images of the designated measurement area were acquired using drones; the designated measurement area included urban, suburban, and rural areas.

[0008] A 3D model is reconstructed from aerial images to obtain a 3D dataset.

[0009] Semantic labeling is performed on the 3D dataset to obtain a 3D dataset with semantic labels.

[0010] Semantic annotation is performed on aerial images to obtain a two-dimensional image dataset with semantic labels.

[0011] Based on the three-dimensional image dataset and the two-dimensional image dataset with semantic labels, a benchmark dataset for urban-scale semantic segmentation is obtained.

[0012] In one embodiment, a drone is used to acquire high-resolution aerial images of a designated measurement area, including:

[0013] The aerial photography mission for the designated measurement area is divided into multiple independent parallel flight missions.

[0014] Each drone performs an independent flight mission according to a pre-planned flight path, capturing images within a specified time interval of 2 seconds; the drone is equipped with a three-axis gimbal and a five-lens tilt photography camera for two-dimensional aerial image acquisition.

[0015] Based on the aerial data captured by all the drones performing the mission, high-resolution aerial images of the designated measurement area are obtained.

[0016] In one embodiment, the high-precision real-time dynamic positioning system is RTK or GNSS.

[0017] In one embodiment, a 3D model is reconstructed based on aerial images to obtain a 3D dataset, including:

[0018] Aerial triangulation is performed on aerial images to determine the position and attitude of the camera and minimize projection errors.

[0019] Based on the results of aerial triangulation, a 3D mesh model of the urban scene was obtained using 3D reconstruction technology. Then, for the large lakes in the scene that were not reconstructed, a semi-automatic water body restoration technology was used to obtain a complete 3D mesh dataset.

[0020] Downsampling of the 3D mesh dataset yields the original point cloud without color information.

[0021] By using nearest neighbor search, the real scene color information on the mesh data texture is transferred to the vertices of the point cloud data, resulting in a point cloud dataset with real color information.

[0022] A 3D dataset is obtained based on the 3D mesh dataset and the point cloud dataset.

[0023] In one embodiment, the 3D dataset is semantically labeled to obtain a 3D dataset with semantic labels, including:

[0024] Follow a consistent semantic category definition standard for annotating 3D meshes and 3D point cloud data.

[0025] Based on the standard definition of semantic categories, 10 semantic categories were identified in the 3D dataset; the 10 semantic categories are: building, road, car, grassland, tall vegetation, playground, lake, farmland, construction site, and ground.

[0026] Annotation tools were used based on semantic categories. Each 3D point or triangular mesh in the 3D dataset was assigned a unique semantic label. All annotated data underwent two rounds of manual cross-checking, resulting in a 3D dataset with semantic labels.

[0027] In one embodiment, the aerial images are semantically annotated to obtain a two-dimensional image dataset with semantic labels, including:

[0028] An automated instance segmentation tool based on the SAM model is used to perform intelligent semantic annotation on aerial images, resulting in a two-dimensional image dataset with semantic labels.

[0029] In one embodiment, the method further includes a three-dimensional mesh data evaluation step:

[0030] Calculate the surface area occupied by triangles in a 3D mesh data.

[0031] For each semantic category, calculate the total surface area of ​​the corresponding semantic category in the 3D mesh dataset to represent the distribution of the semantic category.

[0032] In one embodiment, the method further includes a three-dimensional dataset evaluation step:

[0033] A high-precision real-time dynamic positioning system was used to georeference the aerial images, and six ground control points were manually set in the actual scene.

[0034] The accuracy and reliability of the 3D dataset are evaluated based on the median reprojection error of the first pixel, the median reprojection error of the second pixel of each ground control point, and a preset reasonable error range.

[0035] A device for constructing a city-scale semantic segmentation benchmark dataset, the device comprising:

[0036] The aerial image acquisition module is used to acquire high-resolution aerial images of a designated measurement area using a drone; the designated measurement area includes urban, suburban and rural areas.

[0037] The 3D dataset reconstruction module is used to reconstruct 3D models from aerial images to obtain 3D datasets.

[0038] The semantic annotation module for 3D image datasets is used to perform semantic labeling on 3D datasets, resulting in 3D image datasets with semantic labels.

[0039] The semantic annotation module for two-dimensional image datasets is used to perform semantic annotation on aerial images, resulting in two-dimensional image datasets with semantic labels.

[0040] The module for building a benchmark dataset for urban-scale semantic segmentation is used to obtain a benchmark dataset for urban-scale semantic segmentation based on a 3D image dataset and a 2D image dataset with semantic labels.

[0041] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of any of the methods described above.

[0042] The aforementioned method, apparatus, and equipment for constructing a city-scale semantic segmentation benchmark dataset are described above. The city-scale semantic segmentation benchmark dataset constructed using this method is a large-scale outdoor scene dataset at the city level, containing various data types and rich semantic labels. This dataset includes two types of 3D data: point clouds and meshes, covering an accurately annotated region of 2.85 square kilometers. It includes multiple semantic labels covering semantic information of both urban and rural scenes, serving as a benchmark for 3D semantic segmentation. The dataset also provides raw 2D images captured by UAVs, accompanied by detailed semantic labels. These images can be used in research fields such as 3D reconstruction and high-resolution aerial image semantic segmentation. The dataset demonstrates its applicability to different semantic segmentation baselines. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating a method for constructing a benchmark dataset for urban-scale semantic segmentation in one embodiment.

[0044] Figure 2The following are details of a reconstructed mesh and point cloud scene in one embodiment, where (a) is a textured 3D reconstructed mesh; (b) is a textureless mesh with a white base model; and (c) is a 3D point cloud reconstructed through sampling.

[0045] Figure 3 This is a schematic diagram of semantic annotation of the CUS3D mesh benchmark dataset in one embodiment;

[0046] Figure 4 In another embodiment, a wall with incorrect markings was found and adjusted during manual cross-checking, where (a) the wall with incorrect markings was found, (b) the object was selected using a lasso tool, and (c) the wall with incorrect markings was assigned the correct markings.

[0047] Figure 5 In another embodiment, the distribution of the training set, test set, and validation set is shown, where (a) is the distribution of the training set, test set, and validation set, and (b) is the distribution of semantic categories in the training set (including the validation set) and the test set.

[0048] Figure 6 This is a structural block diagram of a device for constructing a city-scale semantic segmentation benchmark dataset in one embodiment.

[0049] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] The CUS3D dataset is a benchmark dataset for urban-scale semantic segmentation.

[0052] In one embodiment, such as Figure 1 As shown, a method for constructing a city-scale semantic segmentation benchmark dataset is provided, which includes the following steps:

[0053] Step 100: Use a drone to acquire high-resolution aerial images of the designated measurement area; the designated measurement area includes urban, suburban and rural areas.

[0054] Specifically, in existing technologies, acquiring 3D data (such as point clouds) is typically done using a LiDAR scanner on a vehicle. With the development of drone oblique photography and 3D reconstruction technologies, drones have significant advantages in mapping. To acquire high-resolution aerial image sequences, a DJI 300RTK drone can be selected, but is not limited to, equipped with, but is not limited to, the advanced five-camera PSDK102SV3, which is used for 2D aerial image acquisition.

[0055] To ensure comprehensive coverage of the designated measurement area, the drone captures images at pre-planned time intervals of 2 seconds, following a pre-planned flight path. The flight control system is automated. Due to the limited capacity of the drone's onboard batteries, each battery pack supports approximately 30 minutes of flight time. The acquisition of images of the entire area consists of multiple independent, parallel flight missions.

[0056] The designated measurement areas include urban, suburban, and rural areas to meet the needs of smart city construction and urban and rural planning.

[0057] Step 102: Reconstruct a 3D model based on the aerial images to obtain a 3D dataset.

[0058] Specifically, in the reconstruction process of mesh data, the first step is to perform aerial triangulation on the collected 2D image data (aerial images) to determine the camera's position and orientation and minimize projection errors. Then, using the results of the aerial triangulation, a large-scale 3D scene is constructed using 3D reconstruction techniques. However, the CUS3D dataset constructed in this application contains large lakes, whose 3D water bodies do not conform to the Lambert assumption during reconstruction. To address this issue, a semi-automatic water body restoration technique is employed during the mesh data reconstruction process to obtain a complete 3D mesh model; during the point cloud data reconstruction process, the mesh data is downsampled to obtain point cloud data.

[0059] Step 104: Perform semantic labeling on the 3D dataset to obtain a 3D dataset with semantic labels.

[0060] Specifically, the CUS3D dataset contains various data types and diverse semantic information. It includes not only semantic labels for urban scenes but also semantic labels for rural scenes, such as farmland landscapes and suburban infrastructure.

[0061] With the continuous development of modern cities and the expansion of smart city initiatives, incorporating urban and suburban planning and construction into smart city development is becoming increasingly important. By integrating semantic information from rural scenarios into city-level 3D models, accurate and informed decision-making can be provided for smart city planning.

[0062] Step 106: Perform semantic annotation on the aerial images to obtain a two-dimensional image dataset with semantic labels.

[0063] Specifically, in addition to providing 3D mesh and point cloud data with rich semantic labels, the CUS3D dataset also includes detailed semantic annotations for high-resolution raw aerial images taken by drones. Current 2D image semantic segmentation models (such as Deeplab V3+ (Chenetal., 2018)) still have room for improvement in semantic segmentation of large-scale urban aerial images. The semantically labeled 2D aerial images we provide can be used as training data to improve the semantic segmentation performance and accuracy of 2D models such as Deeplab V3+ (Chenetal., 2018) in outdoor urban scenes.

[0064] Step 108: Obtain the city-scale semantic segmentation benchmark dataset based on the 3D image dataset and the 2D image dataset with semantic labels.

[0065] This comprehensive urban-scale semantic segmentation benchmark dataset provides precise, detailed 3D annotations of point cloud and mesh data types, as well as high-resolution raw 2D images with detailed 2D semantic annotations. This dataset is ideal for exploring various semantic segmentation methods. Covering a vast area of ​​approximately 2.85 square kilometers, the dataset contains 10 semantic labels encompassing urban and rural scenes. Each 3D point or triangular mesh in the dataset is meticulously annotated, belonging to one of ten semantic categories.

[0066] In the construction of the aforementioned urban-scale semantic segmentation benchmark dataset, the urban-scale semantic segmentation benchmark dataset constructed using this method is a large-scale outdoor scene dataset at the city level, containing various data types and rich semantic labels. This dataset includes two types of 3D data: point clouds and meshes, covering an accurately annotated region of 2.85 square kilometers. It includes multiple semantic labels covering semantic information of both urban and rural scenes, serving as a benchmark for 3D semantic segmentation. The dataset also provides raw 2D images captured by UAVs, accompanied by detailed semantic labels. These images can be used in research fields such as 3D reconstruction and high-resolution aerial image semantic segmentation, demonstrating the applicability of this dataset to different semantic segmentation baselines.

[0067] In one embodiment, step 100 includes: dividing the aerial photography task of the designated measurement area into multiple independent parallel flight tasks; each drone performs an independent flight task according to a pre-planned flight path and captures images within a specified time interval of 2 seconds; the drone is equipped with a three-axis gimbal five-lens tilt photography camera for two-dimensional aerial image acquisition; and obtaining high-resolution aerial images of the designated measurement area based on the aerial photography data of all drones performing the tasks.

[0068] In one embodiment, step 102 includes: performing aerial triangulation on the aerial image to determine the position and attitude of the camera and minimize projection error; using 3D reconstruction technology to obtain a 3D mesh model of the urban scene based on the results of the aerial triangulation, and then using semi-automatic water body restoration technology for large lakes in the scene that have not been reconstructed to obtain a complete 3D mesh dataset; downsampling the 3D mesh dataset to obtain the original point cloud without color information; transferring the real scene color information on the mesh data texture to the vertices of the point cloud data through nearest neighbor search to obtain a point cloud dataset with real color information; and obtaining a 3D dataset based on the 3D mesh dataset and the point cloud dataset.

[0069] Specifically, the details of the reconstructed mesh and point cloud scene are as follows: Figure 2 As shown, (a) is a textured 3D reconstructed mesh; (b) is a textureless mesh with a white base model; and (c) is a 3D point cloud reconstructed through sampling.

[0070] In the reconstruction of the mesh data, firstly, a large-scale 3D scene was constructed using 3D reconstruction technology based on aerial images. Then, for large lakes in the scene that were not reconstructed, a semi-automatic water body restoration technique was used to restore them, resulting in a complete 3D mesh model. The reconstructed CUS3D mesh dataset was divided into 93 blocks, containing a total of 289,404,088 triangular meshes, covering an area of ​​approximately 2.85 square kilometers.

[0071] During the point cloud data reconstruction process, we downsampled the grid data to obtain the point cloud data. Through nearest neighbor search, we transferred the true color information to the vertices of the point cloud data. The generated dense color point cloud contains 152,298,756 3D points. The point cloud density is approximately 0.15 meters, covering an area of ​​approximately 2.85 square kilometers.

[0072] In one embodiment, step 104 includes: following a consistent semantic category definition standard for annotating 3D meshes and 3D point cloud data; determining 10 semantic categories in the 3D dataset according to the standard definition of semantic categories; the 10 semantic categories are: building, road, car, grassland, tall vegetation, playground, lake, farmland, construction site, and ground; using annotation tools according to semantic categories, assigning a unique semantic label to each 3D point or triangular mesh in the 3D dataset, and all annotation data undergoing two rounds of manual cross-checking; obtaining a 3D dataset with semantic labels.

[0073] Specifically, our CUS3D dataset contains various data types and diverse semantic information. It includes not only semantic labels for urban scenes but also semantic labels for rural scenes, such as farmland landscapes and suburban infrastructure. With the continuous development of modern cities and the expansion of smart city initiatives, integrating urban and suburban planning and construction into smart city development is becoming increasingly important. By integrating the semantic information of rural scenes into city-level 3D models, accurate and informed decision-making can be provided for smart city planning.

[0074] The CUS3D dataset follows a consistent semantic category definition standard for annotating 3D meshes and 3D point cloud data. Each category has a well-defined geometric shape and texture appearance, and each semantic category has a unique meaning consistent with social or academic purposes. The semantics between different categories should be clearly distinguishable. Based on the standard definition of semantic categories, we identified 10 meaningful semantic categories in the CUS3D dataset. Using annotation tools, each 3D point or triangular mesh in the dataset was assigned a unique semantic label. Furthermore, all annotated data underwent two rounds of manual cross-checking. Figure 3 This illustrates a semantic annotation diagram of the CUS3D mesh benchmark dataset. Figure 4 The images show errors that were found and corrected in a timely manner during manual cross-checking, where (a) shows a wall with an incorrect label, (b) shows an object selected using the lasso tool, and (c) shows a wall with an incorrect label being assigned the correct label. Figure 4 The elliptical markers indicate walls where incorrect markings were found. The CUS3D dataset was compared with other existing 3D city datasets. Table 1 shows a comparison of existing 3D benchmark datasets.

[0075] Table 1 Comparison of Existing 3D Benchmark Datasets

[0076]

[0077]

[0078] The CUS3D dataset in this application contains 10 specific object categories as follows: Buildings: including commercial buildings and residences; Roads: including streets, asphalt roads, non-motorized vehicle lanes, and parking lots; Vehicles: including cars and buses; Grasslands: including lawns and shrubs; Tall vegetation: including fir trees and banyan trees; Playgrounds: including basketball courts, athletic fields, and playgrounds; Lakes: including lakes and canals; Farmland: including arable land, undeveloped land, and livestock pens; Construction sites: including building material storage yards and construction sites; Ground:

[0079] This includes cement surfaces, asphalt surfaces, and bare soil surfaces.

[0080] In one embodiment, step 108 includes: using an automated instance segmentation tool based on the SAM model to perform intelligent semantic annotation on the aerial images to obtain a two-dimensional image dataset with semantic labels.

[0081] Specifically, we used an automated instance segmentation tool based on the SAM (Kirillovetal., 2023) model for intelligent semantic annotation. This technique is not limited to specific scenes and can shorten the annotation cycle. We provided 4336 2D image datasets with semantic labels, divided into training, test, and validation sets in an 8:1:1 ratio. Compared to the 3D model, the semantic labels in these images are richer and more detailed, covering 18 different semantic categories of urban scenes, including buildings, lakes, roads, and pedestrians.

[0082] In one embodiment, the method further includes a three-dimensional mesh data evaluation step: calculating the surface area occupied by triangles in the three-dimensional mesh data; and for each semantic category, calculating the total surface area of ​​the corresponding semantic category in the three-dimensional mesh dataset to represent the distribution of the semantic category.

[0083] In one embodiment, the method further includes a three-dimensional dataset evaluation step: using a high-precision real-time dynamic positioning system to georeference the aerial images, and manually setting six ground control points in the actual scene; evaluating the accuracy and reliability of the three-dimensional dataset based on the median reprojection error of the first pixel, the median reprojection error of the second pixel of each ground control point, and a preset reasonable error range.

[0084] In one embodiment, the high-precision real-time dynamic positioning system is RTK or GNSS.

[0085] Specifically, to verify the accuracy and quality of the evaluation data, we used a high-precision real-time dynamic positioning system (RTK GNSS) to georeference the acquired aerial images. In addition, six ground control points were manually set up in the actual scene. After completing the 3D model construction, we calculated the median reprojection error of the first pixel and the median reprojection error of the second pixel at each control point; this error range was within a reasonable range. The first and second pixels are data automatically calculated during the system reconstruction process, and may be, but is not limited to, the first pixel being 0.82 pixels and the second pixel being 0.62 pixels. Therefore, we can consider the constructed 3D model to be accurate and reliable.

[0086] In a validation embodiment, to use the dataset as a benchmark for the semantic segmentation task, we divided all blocks in the CUS3D dataset in an 8:1:1 ratio. 66 blocks were used as training data, 8 blocks as test data, and 8 blocks as validation data (see [link to validation dataset]). Figure 5Because triangles in 3D mesh data vary in size, when evaluating the semantic information of the mesh data, we calculate the surface area occupied by the triangles, rather than the number of triangles. For each semantic category, we calculate the total surface area of ​​the corresponding category in the dataset to represent its distribution, such as... Figure 5 As shown, (a) represents the distribution of the training set, test set, and validation set, and (b) represents the semantic category distribution in the training set (including the validation set) and the test set.

[0087] The category distribution map reveals that some categories, such as cars and playgrounds, account for less than 5% of the total surface area, while categories like buildings, lakes, and tall vegetation account for over 70%. This imbalanced category distribution poses a significant challenge to supervised learning-based semantic segmentation.

[0088] (1) Baseline

[0089] Currently, no open-source deep learning framework can directly process large-scale textured mesh data in urban 3D space. However, in urban environments, point clouds and meshes have almost identical fixed properties. These two types of data share the same set of feature vectors in the same scene. Therefore, we chose point cloud data from our dataset as the input for testing our deep neural network. Test results show that accuracy remains consistent for both types of 3D data. We selected six representative methods—PointNet, PointNet++, RandLA-Net, KPConv, SPGraph, and SQN—as benchmarks for the CUS3D dataset, including supervised and weakly supervised semantic segmentation methods. PointNet is a pioneering work for directly processing unordered point clouds; PointNet++ is an improved version of PointNet, including the extraction of local features; RandLA-Net directly handles semantic segmentation of large-scale point clouds and ranked first in the Semantic3D dataset challenge; KPConv references kernel point convolution for semantic segmentation and achieved good performance on the DALES dataset; SPGraph is one of the methods that uses super point graphs to process large-scale point cloud data; SQN is a weakly supervised method that uses only a small number of labels for semantic segmentation of large-scale point clouds.

[0090] (2) Evaluation Indicators

[0091] For the evaluation metrics of semantic segmentation, we selected precision, recall, F1 score, and intersection-over-union (IoU) for each class, similar to existing benchmark tests. For the overall region test evaluation, we used overall precision (OA), mean precision (mAcc), and mean intersection-over-union (mIoU) as the metrics.

[0092] (3) Benchmark Results

[0093] To ensure fairness, we maintained the original experimental environment for each baseline and conducted experiments on an NVIDIA Tesla V100 GPU. The hyperparameters of each experimental network were adjusted based on validation data to achieve the best possible results. Table 2 lists the quantitative results of the different network experiments after testing. On our dataset benchmark, KPConv performed best across all overall performance metrics, with mIoU of 59.72, OA of 89.42, and mAcc of 97.88. KPConv's good overall and classification performance is attributed to its strong ability to learn local features and capture geometric and semantic information from the data. SQN performed well overall, especially in categories with fewer points (such as cars, sports fields, and construction sites), thanks to its use of shell-based representations for point cloud data, which exhibits good rotation and scaling invariance. Meanwhile, the results show that our dataset performs poorly on PointNet++, primarily because it cannot effectively capture local feature information from aggregated data in large scenes, posing a particular challenge to the widespread application of the CUS3D dataset.

[0094] Table 2 Benchmark results on the CUS3D dataset.

[0095]

[0096] Compared to large-scale 3D datasets collected by LiDAR, the CUS3D dataset provides networks with additional color information, which helps in better understanding the real world. However, this may lead to overfitting. To explore the impact of RGB information on the final semantic segmentation results, we selected five baseline networks for comparative experiments. In each baseline comparison experiment, training was performed with either only coordinate information or both coordinate and RGB information as input. Table 3 shows the quantitative experimental results for different feature inputs on the five baselines. We can see that PointNet (Qietal., 2017a), KPConv (Thomasetal., 2019), and RandLA-Net (Huetal., 2020) achieved better performance when both coordinate and color information were input. Conversely, if only coordinate information was used as input, many city categories could not be correctly identified, such as sites, construction sites, and cars, which had poor segmentation performance. The segmentation performance of SPGraph (Landrieu & Simonovsky, 2018) mainly depends on geometric partitioning, so the availability of RGB information has little impact on its segmentation performance.

[0097] Table 3 shows the quantitative results for five selected baseline models on the CUS3D dataset.

[0098]

[0099]

[0100] The benchmark test results above show that the CUS3D dataset can serve as a benchmark for existing 3D semantic segmentation networks, helping machines to better understand 3D scene information.

[0101] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0102] In one embodiment, such as Figure 6 As shown, a device for constructing a benchmark dataset for urban-scale semantic segmentation is provided, comprising: an aerial image acquisition module, a 3D dataset reconstruction module, a 3D image dataset semantic annotation module, a 2D image dataset semantic annotation module, and an urban-scale semantic segmentation benchmark dataset construction module, wherein:

[0103] The aerial image acquisition module is used to acquire high-resolution aerial images of a designated measurement area using a drone; the designated measurement area includes urban, suburban and rural areas.

[0104] The 3D dataset reconstruction module is used to reconstruct 3D models from aerial images to obtain 3D datasets.

[0105] The semantic annotation module for 3D image datasets is used to perform semantic labeling on 3D datasets, resulting in 3D image datasets with semantic labels.

[0106] The semantic annotation module for two-dimensional image datasets is used to perform semantic annotation on aerial images, resulting in two-dimensional image datasets with semantic labels.

[0107] The module for building a benchmark dataset for urban-scale semantic segmentation is used to obtain a benchmark dataset for urban-scale semantic segmentation based on a 3D image dataset and a 2D image dataset with semantic labels.

[0108] In one embodiment, the aerial image acquisition module is further configured to divide the aerial photography task of the specified measurement area into multiple independent parallel flight tasks; each drone performs an independent flight task according to a pre-planned flight path and captures images within a specified time interval of 2 seconds; the drone is equipped with a three-axis gimbal five-lens oblique photography camera for two-dimensional aerial image acquisition; and a high-resolution aerial image of the specified measurement area is obtained based on the aerial photography data of all drones performing the tasks.

[0109] In one embodiment, the 3D dataset reconstruction module is further used to perform aerial triangulation on aerial images to determine the position and attitude of the camera and minimize projection errors; based on the results of aerial triangulation, a 3D reconstruction technique is used to obtain a 3D mesh model of the urban scene; then, for large lakes in the scene that have not been reconstructed, a semi-automatic water body restoration technique is used to obtain a complete 3D mesh dataset; the 3D mesh dataset is downsampled to obtain an original point cloud without color information; through nearest neighbor search, the real scene color information on the mesh data texture is transferred to the vertices of the point cloud data to obtain a point cloud dataset with real color information; based on the 3D mesh dataset and the point cloud dataset, a 3D dataset is obtained.

[0110] In one embodiment, the semantic annotation module for the 3D image dataset is further configured to follow a consistent semantic category definition standard for annotating 3D meshes and 3D point cloud data. Based on the standard definition of semantic categories, 10 semantic categories are determined in the 3D dataset. The 10 semantic categories are: building, road, car, grassland, tall vegetation, playground, lake, farmland, construction site, and ground. An annotation tool is used according to the semantic categories, and each 3D point or triangular mesh in the 3D dataset is assigned a unique semantic label. All annotation data undergoes two rounds of manual cross-checking, resulting in a 3D dataset with semantic labels.

[0111] In one embodiment, the two-dimensional image dataset semantic annotation module is also used to perform intelligent semantic annotation on aerial images using an automated instance segmentation tool based on the SAM model, so as to obtain a two-dimensional image dataset with semantic labels.

[0112] In one embodiment, the device further includes a three-dimensional mesh data evaluation module for calculating the surface area occupied by triangles in the three-dimensional mesh data; for each semantic category, it calculates the total surface area of ​​the corresponding semantic category in the three-dimensional mesh dataset to represent the distribution of the semantic category.

[0113] In one embodiment, the device further includes a three-dimensional dataset evaluation module for georeferencing aerial images using a high-precision real-time dynamic positioning system and manually setting six ground control points in the actual scene; and evaluating the accuracy and reliability of the three-dimensional dataset based on the median reprojection error of the first pixel, the median reprojection error of the second pixel of each ground control point, and a preset reasonable error range.

[0114] In one embodiment, the high-precision real-time dynamic positioning system in the two-dimensional image data determination module is RTK or GNSS.

[0115] Specific limitations regarding the apparatus for constructing the city-scale semantic segmentation benchmark dataset can be found in the limitations on the construction method of the city-scale semantic segmentation benchmark dataset mentioned above, and will not be repeated here. Each module in the aforementioned apparatus for constructing the city-scale semantic segmentation benchmark dataset can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0116] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for constructing a city-scale semantic segmentation benchmark dataset. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device casing, or an external keyboard, touchpad, or mouse.

[0117] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0118] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0119] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0120] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for constructing a benchmark dataset for urban-scale semantic segmentation, characterized in that, The method includes: High-resolution aerial images of a designated measurement area are acquired using drones; the designated measurement area includes urban, suburban, and rural areas. A 3D model is reconstructed based on the aerial images to obtain a 3D dataset. The three-dimensional dataset is semantically labeled to obtain a three-dimensional dataset with semantic labels; The aerial images are semantically annotated to obtain a two-dimensional image dataset with semantic labels; Based on the 3D dataset and 2D image dataset with semantic labels, a benchmark dataset for urban-scale semantic segmentation is obtained. Based on the aerial images, a 3D model is reconstructed to obtain a 3D dataset, including: Aerial triangulation is performed on the aerial images to determine the position and attitude of the camera and minimize projection error; Based on the results of aerial triangulation, a 3D mesh model of the urban scene was obtained by using 3D reconstruction technology. Then, for the large lakes in the scene that were not reconstructed, a semi-automatic water body restoration technology was used to obtain a complete 3D mesh dataset. The three-dimensional mesh dataset is downsampled to obtain the original point cloud without color information; By using nearest neighbor search, the real scene color information on the mesh data texture is transferred to the vertices of the point cloud data, resulting in a point cloud dataset with real color information. Based on the 3D mesh dataset and the point cloud dataset, a 3D dataset is obtained; The three-dimensional dataset is semantically labeled to obtain a three-dimensional dataset with semantic labels, including: Follow a consistent semantic category definition standard for annotating 3D mesh and 3D point cloud data; Based on the standard definition of semantic categories, 10 semantic categories were identified in the three-dimensional dataset; the 10 semantic categories are: building, road, car, grassland, tall vegetation, playground, lake, farmland, construction site, and ground. Annotation tools were used based on the semantic categories, and each 3D point or triangular mesh in the 3D dataset was assigned a unique semantic label. All annotated data underwent two rounds of manual cross-checking to obtain a 3D dataset with semantic labels. The aerial images are semantically annotated to obtain a two-dimensional image dataset with semantic labels, including: An automated instance segmentation tool based on the SAM model is used to perform intelligent semantic annotation on the aerial images, resulting in a two-dimensional image dataset with semantic labels.

2. The method according to claim 1, characterized in that, High-resolution aerial images of the designated measurement area are acquired using drones, including: The aerial photography mission for the designated measurement area is divided into multiple independent parallel flight missions; Each drone performs an independent flight mission according to a pre-planned flight path, capturing images within a specified time interval of 2 seconds; the drone is equipped with a three-axis gimbal five-lens tilting camera for two-dimensional aerial image acquisition. Based on the aerial data captured by all the drones performing the mission, high-resolution aerial images of the designated measurement area are obtained.

3. The method according to claim 1, characterized in that, The method also includes a three-dimensional mesh data evaluation step: Calculate the surface area occupied by triangles in a 3D mesh data; For each semantic category, the total surface area of ​​the corresponding semantic category in the 3D mesh dataset is calculated to represent the distribution of each semantic category.

4. The method according to claim 1, characterized in that, The method also includes a three-dimensional dataset evaluation step: A high-precision real-time dynamic positioning system was used to perform georeferencing on the aerial images, and six ground control points were manually set in the actual scene. The accuracy and reliability of the 3D dataset are evaluated based on the median reprojection error of the first pixel, the median reprojection error of the second pixel of each ground control point, and a preset reasonable error range.

5. The method according to claim 4, characterized in that, The high-precision real-time dynamic positioning system is either RTK or GNSS.

6. A device for constructing a benchmark dataset for urban-scale semantic segmentation, characterized in that, The apparatus comprising the method according to any one of claims 1 to 5, wherein the apparatus includes: The aerial image acquisition module is used to acquire high-resolution aerial images of a designated measurement area using a drone; the designated measurement area includes urban, suburban, and rural areas. The 3D dataset reconstruction module is used to reconstruct a 3D model based on the aerial images to obtain a 3D dataset. A 3D dataset semantic annotation module is used to perform semantic label annotation on the 3D dataset to obtain a 3D dataset with semantic labels; A two-dimensional image dataset semantic annotation module is used to perform semantic annotation on the aerial images to obtain a two-dimensional image dataset with semantic labels; The module for building a benchmark dataset for urban-scale semantic segmentation is used to obtain a benchmark dataset for urban-scale semantic segmentation based on a 3D dataset and a 2D image dataset with semantic labels.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 5.

Images