Smart city digital twin three-dimensional scene automatic modeling method and device

Abstract

The invention discloses a smart city digital twin three-dimensional scene automatic modeling method and device. The method comprises the steps of obtaining city three-dimensional point cloud data and a scene image collected by an automatic driving vehicle; performing point cloud segmentation on the urban three-dimensional point cloud data to obtain segmentation point cloud data, and classifying all the segmentation point cloud data; performing surface reconstruction on the segmentation point cloud data of different classifications to obtain a surface model corresponding to each segmentation point cloud data; mapping the scene image to each surface model to obtain target three-dimensional model data; and integrating the target three-dimensional model data into the urban three-dimensional model by using a point cloud registration algorithm to obtain an updated urban three-dimensional model. The acquisition of the three-dimensional model data is realized by the automatic driving vehicle, compared with the existing modeling mode of scanning through an unmanned aerial vehicle and the like needing manual intervention, the modeling time and cost are greatly reduced, the automatic driving vehicle can acquire the three-dimensional model data in real time for updating, and the dynamic updating of the urban three-dimensional model is realized.

Description

Technical Field

[0001] The present invention relates to the field of digital twin technology, and in particular to a method and device for automatic modeling of a smart city digital twin three-dimensional scene. Background Art

[0002] With the development of smart cities and digitalization, digital twin technology plays an important role in the construction of smart cities. Digital twin technology can reflect and predict various changes in the physical world in real time, and can provide decision support for city managers by creating a virtual digital copy of the city.

[0003] However, traditional 3D scene modeling methods usually use laser radar scanning, satellite drones, etc. to scan and model cities. This method is not only complex, labor-intensive and costly, but also the city scene is constantly changing, so it is difficult to dynamically update the 3D scene through modeling in the above way. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method and device for automatic modeling of three-dimensional scenes of digital twins of smart cities, improve the efficiency of three-dimensional scene modeling of digital twins of cities, and realize dynamic updating of the model.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: A method for automatic modeling of a smart city digital twin three-dimensional scene, comprising: Obtain urban 3D point cloud data and scene images collected by autonomous driving vehicles; Performing point cloud segmentation on the three-dimensional point cloud data of the city to obtain segmented point cloud data, and classifying all the segmented point cloud data; Performing surface reconstruction on the segmented point cloud data of different categories to obtain a surface model corresponding to each segmented point cloud data; Mapping the scene image onto each of the surface models to obtain target three-dimensional model data; The target three-dimensional model data is integrated into the city three-dimensional model by using a point cloud registration algorithm to obtain an updated city three-dimensional model.

[0006] In order to solve the above technical problems, another technical solution adopted by the present invention is: A smart city digital twin three-dimensional scene automatic modeling device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: Obtain urban 3D point cloud data and scene images collected by autonomous driving vehicles; Performing point cloud segmentation on the three-dimensional point cloud data of the city to obtain segmented point cloud data, and classifying all the segmented point cloud data; Performing surface reconstruction on the segmented point cloud data of different categories to obtain a surface model corresponding to each segmented point cloud data; Mapping the scene image onto each of the surface models to obtain target three-dimensional model data; The target three-dimensional model data is integrated into the city three-dimensional model by using a point cloud registration algorithm to obtain an updated city three-dimensional model.

[0007] The beneficial effects of the present invention are: utilizing the autonomous driving vehicle to collect the 3D point cloud data and scene images of the surrounding scenes in real time during road driving, and completing the construction of a local 3D model by segmenting, classifying, surface reconstructing and texture mapping the 3D point cloud data and scene images, and then integrating the constructed local 3D model with the original city 3D model to obtain an updated city 3D model. The 3D model data collection is realized by the autonomous driving vehicle, which not only greatly reduces the modeling time and cost compared with the existing modeling methods such as scanning by drones that require manual intervention, but the autonomous driving vehicle can also obtain the 3D model data in real time for updating, thereby realizing dynamic updating of the city 3D model. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 This is a flowchart of the steps of a method for automatic modeling of a smart city digital twin three-dimensional scene in an embodiment of the present invention; Figure 2 It is another step flow chart of a method for automatic modeling of a smart city digital twin three-dimensional scene in an embodiment of the present invention; Figure 3 It is a structural schematic diagram of a smart city digital twin three-dimensional scene automatic modeling device in an embodiment of the present invention. DETAILED DESCRIPTION

[0009] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in conjunction with the implementation modes and the accompanying drawings.

[0010] Please refer to Figure 1 , a method for automatic modeling of a smart city digital twin three-dimensional scene, comprising: Obtain urban 3D point cloud data and scene images collected by autonomous driving vehicles; Performing point cloud segmentation on the three-dimensional point cloud data of the city to obtain segmented point cloud data, and classifying all the segmented point cloud data; Performing surface reconstruction on the segmented point cloud data of different categories to obtain a surface model corresponding to each segmented point cloud data; Mapping the scene image onto each of the surface models to obtain target three-dimensional model data; The target three-dimensional model data is integrated into the city three-dimensional model by using a point cloud registration algorithm to obtain an updated city three-dimensional model.

[0011] From the above description, it can be seen that the beneficial effects of the present invention are: using the autonomous driving vehicle to collect three-dimensional point cloud data and scene images of the surrounding scenes in real time during road driving, and completing the construction of a local three-dimensional model by segmenting, classifying, surface reconstructing and texture mapping the three-dimensional point cloud data and scene images, and then integrating the constructed local three-dimensional model with the original city three-dimensional model to obtain an updated city three-dimensional model. The collection of three-dimensional model data is realized by the autonomous driving vehicle. Compared with the existing modeling methods such as scanning by drones that require manual intervention, it not only greatly reduces the modeling time and cost, but the autonomous driving vehicle can also obtain the three-dimensional model data in real time for updating, thereby realizing dynamic updating of the city three-dimensional model.

[0012] Furthermore, the performing point cloud segmentation on the three-dimensional point cloud data of the city to obtain segmented point cloud data includes: Acquiring feature data of the three-dimensional point cloud data of the city, and segmenting point clouds having the same feature data to obtain the segmented point cloud data; The classifying of all the segmented point cloud data comprises: The classification category corresponding to the segmented point cloud data is predicted and outputted through a neural network according to the feature data of the segmented point cloud data.

[0013] From the above description, it can be seen that by acquiring the feature data of three-dimensional point cloud data, it is possible to accurately segment the three-dimensional point cloud data through the local features and global features of the point cloud data, and to classify the segmented point cloud data based on the features of the point cloud data through a neural network.

[0014] Furthermore, the surface reconstruction of the segmented point cloud data of different categories includes: The segmented point cloud data is reconstructed using Poisson surface reconstruction.

[0015] It can be seen from the above description that by using the Poisson surface reconstruction method, a smooth three-dimensional surface can be generated according to the gradient field of the point cloud data, thereby realizing the surface reconstruction of the point cloud data.

[0016] Furthermore, mapping the scene image onto each of the surface models to obtain target three-dimensional model data includes: Obtaining camera posture information of the autonomous driving vehicle; Matching the scene image with the segmented point cloud data according to the camera posture information to obtain a corresponding relationship between the scene image and the surface model; The scene image is mapped onto the corresponding surface model according to the corresponding relationship.

[0017] It can be seen from the above description that by determining the correspondence between point cloud data and scene images through camera posture information, the texture in the scene image can be accurately mapped onto the surface of the surface model.

[0018] Furthermore, after acquiring the urban three-dimensional point cloud data and scene images collected by the autonomous driving vehicle, the method further includes: Performing denoising and filtering processing on the three-dimensional point cloud data of the city to obtain standard three-dimensional point cloud data; The performing point cloud segmentation on the urban three-dimensional point cloud data to obtain segmented point cloud data comprises: The standard three-dimensional point cloud data is subjected to point cloud segmentation to obtain segmented point cloud data.

[0019] From the above description, it can be seen that by performing denoising and filtering on the three-dimensional point cloud data, the noise points in the three-dimensional point cloud data can be effectively removed, and the original geometric structure of the three-dimensional point cloud data can be maintained while reducing the amount of data.

[0020] Another embodiment of the present invention provides a smart city digital twin three-dimensional scene automatic modeling device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: Obtain urban 3D point cloud data and scene images collected by autonomous driving vehicles; Performing point cloud segmentation on the three-dimensional point cloud data of the city to obtain segmented point cloud data, and classifying all the segmented point cloud data; Performing surface reconstruction on the segmented point cloud data of different categories to obtain a surface model corresponding to each segmented point cloud data; Mapping the scene image onto each of the surface models to obtain target three-dimensional model data; The target three-dimensional model data is integrated into the city three-dimensional model by using a point cloud registration algorithm to obtain an updated city three-dimensional model.

[0021] From the above description, it can be seen that the beneficial effects of the present invention are: using the autonomous driving vehicle to collect three-dimensional point cloud data and scene images of the surrounding scenes in real time during road driving, and completing the construction of a local three-dimensional model by segmenting, classifying, surface reconstructing and texture mapping the three-dimensional point cloud data and scene images, and then integrating the constructed local three-dimensional model with the original city three-dimensional model to obtain an updated city three-dimensional model. The collection of three-dimensional model data is realized by the autonomous driving vehicle. Compared with the existing modeling methods such as scanning by drones that require manual intervention, it not only greatly reduces the modeling time and cost, but the autonomous driving vehicle can also obtain the three-dimensional model data in real time for updating, thereby realizing dynamic updating of the city three-dimensional model.

[0022] Furthermore, the performing point cloud segmentation on the three-dimensional point cloud data of the city to obtain segmented point cloud data includes: Acquiring feature data of the three-dimensional point cloud data of the city, and segmenting point clouds having the same feature data to obtain the segmented point cloud data; The classifying of all the segmented point cloud data comprises: The classification category corresponding to the segmented point cloud data is predicted and outputted through a neural network according to the feature data of the segmented point cloud data.

[0023] From the above description, it can be seen that by acquiring the feature data of three-dimensional point cloud data, it is possible to accurately segment the three-dimensional point cloud data through the local features and global features of the point cloud data, and to classify the segmented point cloud data based on the features of the point cloud data through a neural network.

[0024] Furthermore, the surface reconstruction of the segmented point cloud data of different categories includes: The segmented point cloud data is reconstructed using Poisson surface reconstruction.

[0025] It can be seen from the above description that by using the Poisson surface reconstruction method, a smooth three-dimensional surface can be generated according to the gradient field of the point cloud data, thereby realizing the surface reconstruction of the point cloud data.

[0026] Furthermore, mapping the scene image onto each of the surface models to obtain target three-dimensional model data includes: Obtaining camera posture information of the autonomous driving vehicle; Matching the scene image with the segmented point cloud data according to the camera posture information to obtain a corresponding relationship between the scene image and the surface model; The scene image is mapped onto the corresponding surface model according to the corresponding relationship.

[0027] It can be seen from the above description that by determining the correspondence between point cloud data and scene images through camera posture information, the texture in the scene image can be accurately mapped onto the surface of the surface model.

[0028] Furthermore, after acquiring the urban three-dimensional point cloud data and scene images collected by the autonomous driving vehicle, the method further includes: Performing denoising and filtering processing on the three-dimensional point cloud data of the city to obtain standard three-dimensional point cloud data; The performing point cloud segmentation on the urban three-dimensional point cloud data to obtain segmented point cloud data comprises: The standard three-dimensional point cloud data is subjected to point cloud segmentation to obtain segmented point cloud data.

[0029] From the above description, it can be seen that by performing denoising and filtering on the three-dimensional point cloud data, the noise points in the three-dimensional point cloud data can be effectively removed, and the original geometric structure of the three-dimensional point cloud data can be maintained while reducing the amount of data.

[0030] The method and device for automatic modeling of a smart city digital twin three-dimensional scene provided by the present invention can be applied to the scene of constructing a smart city digital twin three-dimensional model, which is described below through specific implementation methods: Embodiment 1 Please refer to Figure 1 as well as Figure 2 , a method for automatic modeling of a smart city digital twin three-dimensional scene, comprising: S1. Acquire urban three-dimensional point cloud data and scene images collected by autonomous driving vehicles; such as acquiring various road data, such as data in driving scenes including lanes, buildings, and green areas, through sensor equipment, cameras, and lidar and other collection devices on autonomous driving vehicles, and convert the above data into point cloud data.

[0031] S2. Perform point cloud segmentation on the urban three-dimensional point cloud data to obtain segmented point cloud data, and classify all segmented point cloud data. Since the point cloud data collected by LiDAR and other sensors usually contain noise, it is necessary to perform denoising and filtering on the collected three-dimensional point cloud data to obtain standard three-dimensional point cloud data; the specific method is as follows: divide the point cloud by voxels, retain the representative points in each voxel, so as to reduce the amount of data while maintaining the geometric structure; for each point pi, calculate the standard deviation σi of the mean ui of its adjacent points within the radius r, remove the points whose difference with the adjacent mean is greater than kσ, and form a denoised point cloud, which is expressed as: ll pi-ui ll>kσi, thereby obtaining standard three-dimensional point cloud data, and then perform point cloud segmentation based on the standard three-dimensional point cloud data.

[0032] S21. Obtain feature data of urban three-dimensional point cloud data, segment point clouds with the same feature data, and obtain segmented point cloud data; for example, the input point cloud data set is represented as: P = {p1, p2, ..., pn}, each point pi is represented by its coordinates (xi, yi, zi) and feature vector fi; wherein each point pi includes local features and global features, reflecting the local and overall properties of each point in the point cloud; local features focus on the local properties of each point in the point cloud, such as normal vector, curvature, surface roughness and other properties, which are used to identify specific objects and details in the point cloud. For example, ‌normal vector‌: represents the surface direction of the point, which is used for point cloud registration, segmentation and classification. ‌Curvature‌: represents the degree of curvature of the point, which is used for surface analysis and shape recognition. ‌Surface roughness‌: represents the local geometric changes of the point, which is used for texture analysis and surface quality assessment. Global features reflect the overall properties of the point cloud, such as shape, distribution, density, etc. Global features are usually invariant to the scale and rotation of the point cloud, and are suitable for tasks such as point cloud retrieval and recognition. The application of global features in point cloud processing includes: ‌Shape‌: describes the overall shape of the point cloud, which is used for overall structure analysis and recognition. ‌Distribution‌: represents the spatial distribution of the point cloud, which is used for spatial relationship analysis and pattern recognition. ‌Density‌: represents the density distribution of the point cloud, which is used for density analysis and space filling evaluation.

[0033] S22. Predict and output the classification category corresponding to the segmented point cloud data according to the feature data of the segmented point cloud data through a neural network; the specific expression is as follows: ci=argmaxsoftmax(fi), ci is the predicted output category.

[0034] S3. Surface reconstruction is performed on segmented point cloud data of different classifications to obtain a surface model corresponding to each segmented point cloud data. In this embodiment, surface reconstruction is performed on segmented point cloud data by Poisson surface reconstruction. A smooth three-dimensional surface is generated according to the gradient field of point cloud data by means of Poisson surface reconstruction. For Poisson reconstruction, the goal is to reconstruct the surface by minimizing the residual of the Poisson equation. The specific formula is: ▽·v=ρ. v is a field representing the surface normal, and ρ is an observation value representing the point cloud, i.e., the normal information. A surface model can be generated by this equation.

[0035] S4. Map the scene image onto each surface model to obtain target three-dimensional model data. Specifically: after obtaining the camera posture information of the autonomous driving vehicle, match the scene image with the segmented point cloud data according to the camera posture information to obtain the correspondence between the scene image and the surface model, and then map the scene image to the corresponding surface model according to the correspondence.

[0036] The specific correspondence is expressed as: S[uv 1]=K·{R·[xyz 1]+t}, where (u,v) is the pixel coordinate in the image, (x,y,z) is the three-dimensional coordinate in the point cloud, R and t are the rotation matrix and translation vector of the camera respectively, and K is the intrinsic parameter matrix of the camera. Through the above transformation, the texture in the image can be mapped to the surface of the three-dimensional model.

[0037] S5. The target 3D model data is integrated into the city 3D model by using a point cloud registration algorithm to obtain an updated city 3D model. That is, the newly collected point cloud is registered and fused using a point cloud registration algorithm, and the new data is integrated into the existing 3D model. At the same time, the 3D model can be continuously updated by iteratively minimizing the error between the new point cloud and the existing model through the ICP (Iterative Closest Point) algorithm.

[0038] Embodiment 2 Please refer to Figure 3 , a smart city digital twin three-dimensional scene automatic modeling device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the computer program: S1. Obtain the city three-dimensional point cloud data and scene images collected by the autonomous driving vehicle. After obtaining the city three-dimensional point cloud data and scene images collected by the autonomous driving vehicle, it is also necessary to denoise and filter the city three-dimensional point cloud data to obtain de-standardized three-dimensional point cloud data.

[0039] S2, performing point cloud segmentation on the urban three-dimensional point cloud data to obtain segmented point cloud data, and classifying all segmented point cloud data; performing point cloud segmentation on the standard three-dimensional point cloud data obtained in step S1 to obtain segmented point cloud data, and the specific segmentation and classification methods are as follows: S21, acquiring feature data of the urban three-dimensional point cloud data, and segmenting the point clouds having the same feature data to obtain segmented point cloud data; S22. Predicting and outputting the classification category corresponding to the segmented point cloud data based on the feature data of the segmented point cloud data through a neural network.

[0040] S3. Surface reconstruction is performed on the segmented point cloud data of different categories to obtain a surface model corresponding to each segmented point cloud data; in an optional implementation, surface reconstruction is performed on the segmented point cloud data by Poisson surface reconstruction.

[0041] S4, mapping the scene image onto each surface model to obtain target three-dimensional model data; specifically: After obtaining the camera posture information of the autonomous driving vehicle, the scene image is matched with the segmented point cloud data according to the camera posture information to obtain the correspondence between the scene image and the surface model, and then the scene image is mapped to the corresponding surface model according to the correspondence.

[0042] S5. Integrate the target 3D model data into the city 3D model by using a point cloud registration algorithm to obtain an updated city 3D model.

[0043] To sum up, the method and device for automatic modeling of smart city digital twin three-dimensional scenes provided by the present invention utilize the autonomous driving vehicle to collect three-dimensional point cloud data and scene images of the surrounding scenes in real time during road driving, and complete the construction of a local three-dimensional model by segmenting, classifying, surface reconstructing and texture mapping the three-dimensional point cloud data and scene images, and then integrate the constructed local three-dimensional model with the original city three-dimensional model to obtain an updated city three-dimensional model. The collection of three-dimensional model data is realized by the autonomous driving vehicle. Compared with the existing modeling methods such as scanning by drones that require manual intervention, it not only greatly reduces the modeling time and cost, but the autonomous driving vehicle can also obtain the three-dimensional model data in real time for updating, thereby realizing dynamic updating of the city three-dimensional model.

[0044] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for automatic modeling of a three-dimensional scene of a smart city digital twin, characterized in that: include: Obtain urban 3D point cloud data and scene images collected by autonomous driving vehicles; Performing point cloud segmentation on the three-dimensional point cloud data of the city to obtain segmented point cloud data, and classifying all the segmented point cloud data; Performing surface reconstruction on the segmented point cloud data of different categories to obtain a surface model corresponding to each segmented point cloud data; Mapping the scene image onto each of the surface models to obtain target three-dimensional model data; The target three-dimensional model data is integrated into the city three-dimensional model by using a point cloud registration algorithm to obtain an updated city three-dimensional model.

2. The method for automatic modeling of a smart city digital twin three-dimensional scene according to claim 1 is characterized in that: The performing point cloud segmentation on the urban three-dimensional point cloud data to obtain segmented point cloud data comprises: Acquiring feature data of the three-dimensional point cloud data of the city, and segmenting point clouds having the same feature data to obtain the segmented point cloud data; The classifying of all the segmented point cloud data comprises: The classification category corresponding to the segmented point cloud data is predicted and outputted through a neural network according to the feature data of the segmented point cloud data.

3. The method for automatic modeling of a smart city digital twin three-dimensional scene according to claim 1 is characterized in that: The surface reconstruction of the segmented point cloud data of different classifications comprises: The segmented point cloud data is reconstructed using Poisson surface reconstruction.

4. The method for automatic modeling of a smart city digital twin three-dimensional scene according to claim 1, characterized in that: Mapping the scene image onto each of the surface models to obtain target three-dimensional model data comprises: Obtaining camera posture information of the autonomous driving vehicle; Matching the scene image with the segmented point cloud data according to the camera posture information to obtain a corresponding relationship between the scene image and the surface model; The scene image is mapped onto the corresponding surface model according to the corresponding relationship.

5. The method for automatic modeling of a smart city digital twin three-dimensional scene according to claim 1, characterized in that: After obtaining the urban three-dimensional point cloud data and scene images collected by the autonomous driving vehicle, the following further includes: Performing denoising and filtering processing on the three-dimensional point cloud data of the city to obtain standard three-dimensional point cloud data; The performing point cloud segmentation on the urban three-dimensional point cloud data to obtain segmented point cloud data comprises: The standard three-dimensional point cloud data is subjected to point cloud segmentation to obtain segmented point cloud data.

6. A smart city digital twin three-dimensional scene automatic modeling device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the following steps are implemented: Obtain urban 3D point cloud data and scene images collected by autonomous driving vehicles; Performing point cloud segmentation on the three-dimensional point cloud data of the city to obtain segmented point cloud data, and classifying all the segmented point cloud data; Performing surface reconstruction on the segmented point cloud data of different categories to obtain a surface model corresponding to each segmented point cloud data; Mapping the scene image onto each of the surface models to obtain target three-dimensional model data; The target three-dimensional model data is integrated into the city three-dimensional model by using a point cloud registration algorithm to obtain an updated city three-dimensional model.

7. The device for automatic modeling of a three-dimensional scene of a smart city digital twin according to claim 6, characterized in that: The performing point cloud segmentation on the urban three-dimensional point cloud data to obtain segmented point cloud data comprises: Acquiring feature data of the three-dimensional point cloud data of the city, and segmenting point clouds having the same feature data to obtain the segmented point cloud data; The classifying of all the segmented point cloud data comprises: The classification category corresponding to the segmented point cloud data is predicted and outputted through a neural network according to the feature data of the segmented point cloud data.

8. The device for automatic modeling of a smart city digital twin three-dimensional scene according to claim 6, characterized in that: The surface reconstruction of the segmented point cloud data of different classifications comprises: The segmented point cloud data is reconstructed using Poisson surface reconstruction.

9. The device for automatic modeling of a three-dimensional scene of a smart city digital twin according to claim 6, characterized in that: Mapping the scene image onto each of the surface models to obtain target three-dimensional model data comprises: Obtaining camera posture information of the autonomous driving vehicle; Matching the scene image with the segmented point cloud data according to the camera posture information to obtain a corresponding relationship between the scene image and the surface model; The scene image is mapped onto the corresponding surface model according to the corresponding relationship.

10. The device for automatic modeling of a three-dimensional scene of a smart city digital twin according to claim 6, characterized in that: After obtaining the urban three-dimensional point cloud data and scene images collected by the autonomous driving vehicle, the following further includes: Performing denoising and filtering processing on the three-dimensional point cloud data of the city to obtain standard three-dimensional point cloud data; The performing point cloud segmentation on the urban three-dimensional point cloud data to obtain segmented point cloud data comprises: The standard three-dimensional point cloud data is subjected to point cloud segmentation to obtain segmented point cloud data.

Images