Crowdsourced disordered image assisted urban scene reconstruction method and device and storage medium

Abstract

The application relates to a crowd-sourced unordered image auxiliary-based urban scene reconstruction method and device and a storage medium, wherein the method comprises the following steps: acquiring laser radar data to form basic point cloud data; acquiring crowd-sourced unordered images, estimating a depth map by using a depth estimation network, and obtaining auxiliary point cloud data; fusing the basic point cloud data and the auxiliary point cloud data to obtain geometric information; acquiring multispectral images and panchromatic images obtained by satellite remote sensing, determining ground object color information based on the multispectral images, and determining a ground surface texture model and a ground object model based on the panchromatic images; performing data preprocessing on the crowd-sourced unordered images to obtain a texture image, combining the ground object model and the ground object color information to obtain a ground object texture model; determining texture information based on the ground object texture model and the ground surface texture model; and realizing urban scene reconstruction by using an automatic reconstruction solution based on the geometric information and the texture information. Compared with the prior art, the application has the advantages of high restoration degree and high reconstruction precision.

Description

Technical Field

[0001] This invention relates to the field of smart city scene reconstruction, and in particular to a method, apparatus and storage medium for urban scene reconstruction based on crowdsourced disordered images. Background Technology

[0002] Urban scene reconstruction is a hot topic in the fields of computer science and geographic information, and is widely used in smart cities and virtual simulation. Its role in intelligent transportation, intelligent navigation, and smart city management is also becoming increasingly prominent, playing a significant role in promoting the construction of smart cities.

[0003] Currently, the methods used for urban scene reconstruction can be mainly divided into the following categories:

[0004] (1) The city reconstruction method based on lidar obtains target point cloud data in the scene by scanning objects in the urban scene with lidar, which solves the problems of high requirements for acquisition equipment and long processing time in the existing technology and breaks the limitation of resolution limitation. However, at the same time, it lacks the texture information of objects, resulting in low reconstruction accuracy.

[0005] (2) Urban reconstruction method based on aerospace remote sensing: With the development of aerospace remote sensing technology, the low cost and high resolution of aerospace remote sensing have been favored by researchers and applied to urban scene reconstruction, providing more accurate texture, geometry and semantic information for 3D urban modeling, making up for the "lack of spectral information" of lidar technology.

[0006] (3) Urban reconstruction methods based on digital photogrammetry can obtain not only geometric and texture reconstruction information, but also topological and semantic information. In particular, it has high reconstruction accuracy for man-made features with clear outlines and is often used as an aid to urban modeling by using texture information from LiDAR data. However, due to the complexity of objects in urban scenes, the high accuracy of digital photogrammetry relies on multi-angle shooting and is highly dependent on the cooperation of multiple disciplines (communication, image processing, computer vision), which increases the workload and reduces work efficiency.

[0007] (4) The urban reconstruction method based on multi-source disordered imagery mainly utilizes the shared and abundant network resources of the Internet to search for information about relevant scenes. Especially for some complex scenes that require detailed modeling, multi-source disordered imagery is a good supplement in the reconstruction of special urban scenes. However, it also has its own limitations. For example, due to the lack of some accurate information, different shooting locations, and the limitations of the equipment itself, there will be a lot of redundant data when acquiring point cloud data from disordered images, resulting in slow reconstruction speed and poor reconstruction effect. Summary of the Invention

[0008] The purpose of this invention is to provide a method, device, and storage medium for urban scene reconstruction based on mass-source disordered imagery. This method uses lidar to fuse satellite remote sensing and mass-source disordered imagery to reconstruct urban scenes, resulting in low data redundancy, high fidelity, high reconstruction accuracy, and fast reconstruction speed.

[0009] The objective of this invention can be achieved through the following technical solutions:

[0010] A method for urban scene reconstruction based on crowdsourced, disordered imagery includes the following steps:

[0011] Acquire lidar data and form basic point cloud data based on the lidar data;

[0012] The process involves acquiring and filtering mass-source disordered images using web crawling technology, estimating depth maps of the mass-source disordered images using a depth estimation network, and obtaining auxiliary point cloud data based on the depth maps and pose information of the mass-source disordered images. The sources of the mass-source disordered images include urban surveillance and internet sharing.

[0013] Geometric information is obtained by fusing basic point cloud data and auxiliary point cloud data;

[0014] Acquire multispectral and panchromatic images based on satellite remote sensing; determine the color information of ground features based on multispectral images; and determine the surface texture model and ground feature model based on panchromatic images.

[0015] Data preprocessing is performed on unordered images from multiple sources to obtain texture images, and then combined with the ground feature model and ground feature color information to obtain a ground feature texture model;

[0016] Texture information is determined based on land cover texture model and land surface texture model;

[0017] Based on geometric and texture information, an automatic reconstruction solution is used to reconstruct urban scenes and obtain a reconstructed 3D model.

[0018] The lidar data includes raw data and pose data. The raw data includes the three-dimensional coordinates of the laser, as well as scanning angle information, echo count information, and echo intensity information.

[0019] The process of forming basic point cloud data based on lidar data specifically involves: obtaining an original point cloud based on the original data and pose data; processing the original point cloud using flight strip stitching and filtering techniques to obtain an effective point cloud; and using the effective point cloud as the basic point cloud data.

[0020] The input to the depth estimation network is an RGB image from a crowdsourced, unordered video frame. The network model is an encoder-decoder structure, and the output is a depth map, where the value of the depth map is the depth value of each pixel in the RGB image.

[0021] The encoder includes four downsampling modules, and the decoder includes four upsampling modules.

[0022] During the training phase, the depth estimation network uses the RGB image of frame t as the source image and the RGB image of frame t+1 as the target image. Based on the projection relationship, each pixel in the source image is reprojected to obtain the reconstructed target image, thereby determining the depth value. The photometric loss between the reconstructed target image and the real target image is used as the loss function for depth estimation, achieving unsupervised training.

[0023] The projection relationship is as follows:

[0024] p s ~KT t→s D t (p t )K -1 p t

[0025] Where, p s p represents the pixels in the source image. t To reconstruct the pixels of the target image, K is the intrinsic parameter matrix, T is the pose estimation, and D is the depth value;

[0026] The loss function is:

[0027]

[0028] in, To reconstruct the target image, I t For the real target image, SSIM is the image similarity loss, and γ is the weight.

[0029] The data preprocessing includes coordinate transformation, data removal, and filtering. The coordinate transformation includes converting three-dimensional coordinates to two-dimensional geometric coordinates and converting two-dimensional geometric coordinates to texture coordinates based on corner correction.

[0030] The method further includes verifying the texture of the reconstructed 3D model based on verification criteria, wherein the verification criteria include:

[0031] 1) Reflects the true colors and patterns of the building;

[0032] 2) The textures in the same area are consistent and harmonious;

[0033] 3) The texture level matches the level of the geometric information, and the clarity meets the pre-configured standard.

[0034] A device for urban scene reconstruction based on crowdsourced disordered images includes a memory, a processor, and a program stored in the memory. When the processor executes the program, it implements the method described above.

[0035] A storage medium having a program stored thereon, which, when executed, implements the method described above.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] (1) This invention uses point cloud data from lidar to construct urban geometric features, and combines the texture information of artificial features in satellite remote sensing and mass-source disordered images to completely reconstruct urban scenes. The scene reconstruction has a strong sense of realism, high restoration degree, and good reconstruction effect. It can not only restore topographic data, but also restore accurate building texture information, thus solving the problem of information loss in urban scene reconstruction.

[0038] (2) The depth estimation network used in this invention is an end-to-end unsupervised network, which does not require additional supervision signals, reducing the difficulty of utilizing crowdsourced disordered images and improving reconstruction efficiency. Attached Figure Description

[0039] Figure 1 This is a flowchart of the method of the present invention;

[0040] Figure 2 This is a schematic diagram of the lidar data processing flow;

[0041] Figure 3 This is a schematic diagram of the satellite remote sensing image processing workflow.

[0042] Figure 4 A schematic diagram illustrating the process of forming point cloud data;

[0043] Figure 5 This is a diagram illustrating coordinate transformation. Detailed Implementation

[0044] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0045] A method for urban scene reconstruction based on crowdsourced, disordered imagery, such as Figure 1 As shown, it includes the following steps:

[0046] Step 1) Acquire LiDAR data and form basic point cloud data based on the LiDAR data, such as... Figure 2 As shown;

[0047] In this embodiment, a quadcopter drone equipped with a LiteMapper2800 / 4600 laser scanner is used to scan urban scenes and collect lidar data.

[0048] LiDAR data includes raw data and pose data. The raw data includes the laser's three-dimensional coordinates, scanning angle information, echo count information, and echo intensity information.

[0049] Based on the raw data, combined with pose data obtained through GPS and IMU information, the raw point cloud is obtained. Considering the large amount of point cloud data, it is necessary to divide the data into blocks, stitch the point cloud data according to different flight strips, eliminate noise through feature extraction and edge detection algorithms, filter out non-ground points, and obtain the effective point cloud, which is used as the base point cloud data.

[0050] Step 2) Obtain and filter crowdsourced unordered images using web crawling techniques, estimate depth maps of the crowdsourced unordered images using a depth estimation network, and obtain auxiliary point cloud data based on the depth maps and pose information of the crowdsourced unordered images, such as... Figure 4 As shown;

[0051] Crowdsourced unordered images refer to image materials and data collected and made public by the public, such as images and videos taken by electronic devices such as digital cameras, mobile phones, and tablets, surveillance videos in cities, and image information of some special scenes, such as network resources obtained through urban surveillance network cameras, network search engines, and Google Earth platform commonly used for 3D city modeling, including officially released topographic maps, multispectral images, street view images taken by the public, and landmark buildings.

[0052] By using web crawling technology, urban-related scene images and multi-source image data are filtered from mass unordered images to obtain highly relevant urban scene information and form a data set.

[0053] The depth estimation network takes RGB images from unordered video frames as input, uses an encoder-decoder structure as its network model, and outputs a depth map, where the depth value is the depth value of each pixel in the RGB image. The encoder includes four downsampling modules, and the decoder includes four upsampling modules.

[0054] During the training phase, the depth estimation network uses the RGB image of frame t as the source image and the RGB image of frame t+1 as the target image. Based on the projection relationship, each pixel in the source image is reprojected to obtain the reconstructed target image, thereby determining the depth value. The photometric loss between the reconstructed target image and the ground truth target image is used as the loss function for depth estimation, achieving unsupervised training.

[0055] The projection relationship is as follows:

[0056] p s ~KT t→s D t (p t )K -1p t

[0057] Where, p s p represents the pixels in the source image. t To reconstruct the pixels of the target image, K is the intrinsic parameter matrix, T is the pose estimation, and D is the depth value;

[0058] The loss function is:

[0059]

[0060] in, To reconstruct the target image, I t For the real target image, SSIM is the image similarity loss, and γ is the weight.

[0061] The above method can recover pixel-level depth maps from a single image in an end-to-end manner, and then use pose information to generate auxiliary point cloud data.

[0062] Step 3) Fuse the basic point cloud data and the auxiliary point cloud data to obtain geometric information;

[0063] Auxiliary point cloud data obtained from unordered images from multiple sources can be used in conjunction with LiDAR to reconstruct 3D models with higher accuracy.

[0064] Step 4) Acquire multispectral and panchromatic images based on satellite remote sensing. Determine the color information of ground features based on the multispectral images, and determine the surface texture model and ground feature model based on the panchromatic images, such as... Figure 3 As shown;

[0065] Multispectral and panchromatic images obtained from satellite remote sensing are mainly acquired through open network platforms, including satellite remote sensing images, topographic data, and commercial satellite remote sensing images. These platforms include domestic ones such as the "Geospatial Data Cloud," the "Institute of Remote Sensing and Digital Earth, Chinese Academy of Sciences," the "National Geographic Information Resources Catalog Service System," the "National Earth System Science Data Center," and the "Earth Big Data Science and Engineering Data Sharing Service System," as well as international platforms such as the "United States Geological Survey."

[0066] In this embodiment, high-resolution panchromatic imagery and low-resolution multispectral imagery are downloaded from the "Geographic Data Cloud" platform. Three-dimensional data is obtained from the images, and a digital elevation model is used to characterize the terrain surface. The orthophoto map of the satellite imagery generated from the panchromatic imagery is cropped according to geographic coordinates and subjected to a series of transformations such as contrast enhancement. The transformed imagery is used as the texture of the terrain to cover it, resulting in a surface texture model. Spectral information corresponds to color information. Multispectral imagery can be used to obtain the color information of ground features. Combined with panchromatic imagery, high-resolution remote sensing images with color information can be obtained, resulting in a high-resolution ground feature color model.

[0067] Step 5) Perform data preprocessing on the unordered images from multiple sources to obtain texture images, and combine them with the ground feature model and ground feature color information to obtain the ground feature texture model;

[0068] Before performing data preprocessing on the disordered images from multiple sources, precise screening is performed by registering pose information to remove images that do not meet the requirements, such as those that are blurry or occluded.

[0069] The data preprocessing includes coordinate transformation, data removal, and filtering. The coordinate transformation includes converting 3D coordinates to 2D geometric coordinates, and converting 2D geometric coordinates to texture coordinates based on corner correction. Figure 5 As shown; the data removal involves removing abnormally collected data and outlier values; then, the data is denoised using a Kalman filter to obtain more accurate texture information.

[0070] Step 6) Determine texture information based on the land cover texture model and the land surface texture model;

[0071] Step 7) Based on geometric and texture information, use an automatic reconstruction solution to reconstruct the urban scene and obtain a reconstructed 3D model;

[0072] In this embodiment, the automatic reconstruction solution provided by the Environment-for-3D-Exploitation platform is used to achieve scene reconstruction. This software can perform real-world scene browsing, DSM extraction, DEM extraction, extraction of features such as trees, buildings, and power lines, and directly reconstruct a 3D model from experimental data.

[0073] Step 8) Verify the texture of the reconstructed 3D model based on verification criteria, wherein the verification criteria include:

[0074] 1) Reflects the true colors and patterns of the building;

[0075] 2) The textures in the same area are consistent and harmonious;

[0076] 3) The texture level matches the level of the geometric information, and the clarity meets the pre-configured standard.

[0077] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0078] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for urban scene reconstruction based on crowdsourced unordered images, characterized in that, Includes the following steps: Acquire lidar data and form basic point cloud data based on the lidar data; The process involves acquiring and filtering mass-source disordered images using web crawling technology, estimating depth maps of the mass-source disordered images using a depth estimation network, and obtaining auxiliary point cloud data based on the depth maps and pose information of the mass-source disordered images. The sources of the mass-source disordered images include urban surveillance and internet sharing. Geometric information is obtained by fusing basic point cloud data and auxiliary point cloud data; Acquire multispectral and panchromatic images based on satellite remote sensing; determine the color information of ground features based on multispectral images; and determine the surface texture model and ground feature model based on panchromatic images. Data preprocessing is performed on unordered images from multiple sources to obtain texture images, and then combined with the ground feature model and ground feature color information to obtain a ground feature texture model; Texture information is determined based on land cover texture model and land surface texture model; Based on geometric and texture information, an automatic reconstruction solution is used to reconstruct urban scenes and obtain a reconstructed 3D model. The input to the depth estimation network is an RGB image from a crowdsourced, unordered video frame. The network model is an encoder-decoder structure, and the output is a depth map. The value of the depth map is the depth value of each pixel in the RGB image. The encoder includes four downsampling modules, and the decoder includes four upsampling modules.

2. The urban scene reconstruction method based on crowdsourced disordered imagery as described in claim 1, characterized in that, The lidar data includes raw data and pose data. The raw data includes the three-dimensional coordinates of the laser, as well as scanning angle information, echo count information, and echo intensity information.

3. The urban scene reconstruction method based on crowdsourced disordered imagery as described in claim 2, characterized in that, The process of forming basic point cloud data based on lidar data specifically involves: obtaining an original point cloud based on the original data and pose data; processing the original point cloud using flight strip stitching and filtering techniques to obtain an effective point cloud; and using the effective point cloud as the basic point cloud data.

4. The urban scene reconstruction method based on crowdsourced disordered imagery as described in claim 1, characterized in that, During the training phase, the depth estimation network uses the RGB image of frame t as the source image and the RGB image of frame t+1 as the target image. Based on the projection relationship, each pixel in the source image is reprojected to obtain the reconstructed target image, thereby determining the depth value. The photometric loss between the reconstructed target image and the real target image is used as the loss function for depth estimation, achieving unsupervised training. The projection relationship is as follows: in, For pixels in the source image, To reconstruct the pixels of the target image, The intrinsic parameter matrix, For pose estimation, This is the depth value; The loss function is: in, To reconstruct the target image, For the real target image, SSIM is the image similarity loss. For weights.

5. The urban scene reconstruction method based on crowdsourced disordered imagery as described in claim 1, characterized in that, The data preprocessing includes coordinate transformation, data removal, and filtering. The coordinate transformation includes converting three-dimensional coordinates to two-dimensional geometric coordinates and converting two-dimensional geometric coordinates to texture coordinates based on corner correction.

6. The urban scene reconstruction method based on crowdsourced disordered imagery as described in claim 1, characterized in that, The method further includes verifying the texture of the reconstructed 3D model based on verification criteria, wherein the verification criteria include: 1) Reflects the true colors and patterns of the building; 2) The textures in the same area are consistent and harmonious; 3) The texture level matches the level of the geometric information, and the clarity meets the pre-configured standard.

7. A device for urban scene reconstruction based on crowdsourced disordered imagery, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-6.

8. A storage medium having a program stored thereon, characterized in that, When the program is executed, it implements the method as described in any one of claims 1-6.

Images