Method and system for constructing digital twin oblique photography model based on fine modeling

Abstract

The present invention relates to the technical field of three-dimensional city modeling, and specifically to a method and system for constructing a digital twin oblique photography model based on fine modeling. S1. Aerial photography is performed by using a drone equipped with a 5-lens oblique photography device to obtain a three-dimensional point cloud and aerial images of the shooting area; S2. The original three-dimensional point cloud is subjected to voxel discretization processing to obtain a voxelized three-dimensional grid, and the number of point clouds is recorded for each voxel. The three-dimensional occlusion rate index is determined based on the number of points in the voxel, the occlusion area in the shooting area is identified, and the occlusion grade of the occlusion area is determined; S3. A two-dimensional image of the occlusion area is obtained through aerial images, and the point cloud density of pixels and associated voxels at different time sequences is calculated to determine the occlusion type; S4. According to the occlusion grade and occlusion type of the occlusion area, it is identified whether the occlusion area is repairable; S5. Based on the positions of all irreparable occlusion areas, a collection route for a ground mobile collection platform is generated.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional city modeling, and in particular to a method and system for constructing a digital twin oblique photography model based on fine modeling. Background Art

[0002] 3D City Modeling (3DCM) is a three-dimensional model created based on two-dimensional geographic information. It digitally simulates urban terrain and features in three dimensions, creating a virtual city environment similar to real life, providing three-dimensional information services for urban planning, construction, and operational decision-making. Through program development, it has evolved into a three-dimensional geographic information system, which can be used to analyze a city's natural and constructed elements. Through interactive operations, users can gain a realistic and intuitive experience of the virtual city environment.

[0003] Oblique photography is a high-tech technology developed in the international photogrammetry field over the past decade. By simultaneously capturing images from five different perspectives—one vertical, one oblique, and four tilted—this technology captures rich, high-resolution textures of building tops and side views. This technology not only accurately reflects the terrain and captures object-space texture information, but also generates realistic 3D city models through positioning, fusion, and modeling techniques. While oblique photography can solve the problem of large-scale 3D urban modeling, it can also create blind spots due to obstructions such as eaves, awnings, and narrow alleyways. Therefore, while oblique photography is now used for aerial photography, ground-based supplementary photography is also used for coordination. For example, the patent application number CN201620939472.8, "Air-ground integrated system for obtaining digital city real-life 3D modeling data", discloses that by organically integrating aerial oblique photography measurement with ground mobile vehicle measurement, the shortcomings of aerial oblique photography near the ground can be compensated to the greatest extent, and the advantages of ground mobile vehicle measurement near the ground can be fully utilized, thereby quickly obtaining real, full-factor, complete and clear details, high-precision and high-quality urban 3D modeling data, making an important contribution to the rapid production of digital city real-life 3D models.

[0004] However, this approach relies on manual annotation to repair obstructed areas. After manually inspecting and annotating the aerial data, ground-based equipment is used to retake the images. This approach is inefficient, requiring approximately 5-8 hours of manual intervention per square kilometer. Furthermore, the dynamic update mechanism is inefficient, making it impossible to update the model in real time as urban elements change. Summary of the Invention

[0005] The technical problem solved by the present invention is to provide a method and system for constructing a digital twin oblique photography model based on fine modeling, which can improve the updating efficiency of urban modeling.

[0006] The basic solution provided by the present invention is a method for constructing a digital twin oblique photography model based on fine modeling, which includes the following steps:

[0007] S1. Use a drone equipped with a 5-lens oblique camera to take aerial photos and obtain a 3D point cloud and aerial images of the shooting area.

[0008] S2. Voxelize and discretize the original 3D point cloud to obtain a voxelized 3D grid. Each voxel records the number of point clouds. The point cloud density of each voxel is calculated based on the number of points within the voxel. The 3D occlusion rate index of each voxel is determined based on the point cloud density of each voxel. Based on the 3D occlusion quantification index of each voxel block, the occluded area in the captured area is identified and the occlusion classification of the occluded area is determined.

[0009] S3. Obtain a two-dimensional image of the occluded area using aerial imagery, map the two-dimensional image to a voxel coordinate system of a three-dimensional point cloud, associate pixels in the two-dimensional image with point cloud densities in voxels of the three-dimensional point cloud, calculate correlations between the point cloud densities of pixels and associated voxels at different time series based on the time series data, and determine the type of occlusion.

[0010] S4. Identify whether the occluded area is repairable based on the occlusion classification and occlusion type of the occluded area. If the occluded area is repairable, select a preset repair algorithm based on the occlusion classification and occlusion type to repair the occluded area. If the occluded area is unrepairable, identify the location of the unrepairable area and store it.

[0011] S5. Generate a collection route for the ground mobile collection platform based on the locations of all the irreparable occlusion areas, and perform ground collection on the irreparable occlusion areas.

[0012] The principles and advantages of the present invention are as follows: Aerial photography is performed using a drone equipped with an oblique camera to collect 3D point clouds and aerial images of various urban areas. The collected raw 3D point clouds are discretized into voxels to produce a voxelized 3D grid, with each voxel containing a number of point clouds. The point cloud density of each voxel is determined by analyzing the number of point clouds within the voxel. If occlusion occurs, the point cloud density within the occluded area will be significantly lower than that of other nearby voxels. Therefore, by identifying the point cloud density within the voxel and the voxel cluster with abnormal point cloud density, the occluded area can be determined. Aerial photography is then used to obtain a 2D image of the occluded area, and the pixels in the 2D image are correlated with the point cloud density of the voxels in the 3D point cloud to identify the occlusion type, which can be static or dynamic. Dynamic occlusion refers to the movement of obstructing objects over time or space, resulting in dynamic changes in the position and shape of the obstructed area. Examples include pedestrians obstructing buildings, vehicles obstructing roads, and birds obstructing drone photography. Static occlusion refers to an obstruction with a fixed position and shape, and the obstructed area remains consistent across multiple shots. For example, trees obscuring facades, billboards obscuring roofs, and bridges obscuring the ground. Dynamic occlusion manifests itself in imagery as temporal changes in pixel values, and in point clouds as abnormal fluctuations in point cloud density within voxels. Static occlusion manifests itself as a fixed texture in imagery and as stable, low-density voxels in point clouds. Therefore, by correlating the two, we can identify whether the obstructed area is static or dynamic.

[0013] After identifying the occlusion type and occlusion classification, the system determines whether the occlusion is repairable based on the occlusion classification and type. If it is repairable, the system uses a pre-set repair algorithm to repair the occlusion based on the occlusion classification and type. If it is unrepairable, the location of the unrepairable occlusion is stored and recorded, and a collection route for the ground mobile collection platform is generated based on the existing location of the unrepairable occlusion area to perform ground-based re-shooting of the unrepairable occlusion area.

[0014] Compared with the existing technology, in this application, the occlusion classification and occlusion type of the occlusion area are identified. According to the occlusion classification and occlusion type of the occlusion area, it is determined whether the occlusion area can be self-repaired through algorithms such as GAN adversarial network and calling BIM data completion. If it can be self-repaired, the corresponding algorithm can be called for automatic repair. If it cannot be self-repaired, the occluded area is re-photographed by ground re-shooting. Traditional manual labeling takes 5-8 hours for one square kilometer, and it takes a lot of time to label all areas. Therefore, it is usually necessary to re-shoot the occluded part of an area after marking it, and then mark the next area at the same time. The technical solution in this application can find all the occlusion areas that need to be re-photographed at one time, and plan the optimal re-shooting path according to the location of each occlusion area. Compared with the existing technology, a lot of time is saved in both area labeling and re-shooting planning, so it can be applied to the dynamic update of real-time urban models.

[0015] By correlating the voxel point cloud density in a 3D point cloud with the pixel values in a 2D image, occlusion types can be identified more accurately than with traditional object detection. For example, traditional object detection uses the outline of an object to determine the type of occlusion, potentially misidentifying a human-shaped sculpture as a pedestrian and thus identifying it as dynamic occlusion. However, this application considers only pixel values and point cloud density to determine occlusion type, enabling more precise determinations and providing accurate data for subsequent processing and analysis.

[0016] Furthermore, the step S2 includes the following steps:

[0017] S21, inputting the original 3D point cloud data into an adaptive voxel partitioning algorithm, outputting a voxelized 3D grid, and recording the number of point clouds in each voxel;

[0018] S22. Calculate the point cloud density of each voxel:

[0019]

[0020] in is the voxel density, is the number of point clouds within the voxel, is the voxel volume;

[0021] S23. Calculate the density difference between the current voxel and the 26 neighboring voxels:

[0022]

[0023] in is the average density of the 26 neighborhoods;

[0024] S24. Calculate the visibility coefficient by ray casting algorithm , combined with point cloud density and density differences Calculate the three-dimensional occlusion rate index:

[0025]

[0026] in is the maximum reference value of point cloud density, according to the output three-dimensional occlusion rate index , when the three-dimensional occlusion rate index Above the preset occlusion threshold , the area where the voxel is located is marked as the occlusion area, and the three-dimensional occlusion rate index The higher it is, the higher the occlusion level.

[0027] The point cloud is voxelized and the original point cloud is input into the adaptive voxel partitioning algorithm to obtain a voxelized three-dimensional grid according to the preset voxel specifications, such as a 32*32*32 voxel grid, where each voxel is a cubic unit. Each voxel contains several point clouds. Based on the number of point clouds in the voxel, the point cloud density of the voxel is calculated. Since the occluded area usually forms a local "density depression", the density difference between the current voxel and the surrounding environment is measured through 26 fields. The 26 fields are 6 face fields, including the fields of 6 shared faces (up and down, front and back, left and right); the edge fields in the diagonal direction of 12 shared edges and the corner fields in the diagonal direction of 8 shared vertices, totaling 26 fields. If the density of a voxel is significantly lower than that of its surrounding environment, it is due to data loss caused by occlusion.

[0028] Furthermore, the S2 further includes the following steps:

[0029] S25, identifying the number of abnormal voxels, when the number of abnormal voxels Below the preset quantity threshold And the voxel volume Shrink voxels larger than the preset voxel size lower limit When , adjust the voxel specifications and re-execute S21 to S24;

[0030] S26, when When , the current voxel division is retained and S3 is executed;

[0031] S27, when and When , it is determined that the occlusion is not significant and the adjustment is terminated;

[0032] The S26 and S27 ,in is the initial voxel size.

[0033] A minimum threshold for the number of abnormal voxels and a minimum allowable value for the voxel volume are preset. When the number of abnormal voxels falls below the minimum threshold, dual threshold constraints are used to distinguish between "low-density occlusion" and "non-occluded areas" by using the voxel specification lower limit and the number threshold. Dynamic adjustment strategies are applied to occlusion characteristics in different scenarios, improving algorithm robustness. Setting a minimum allowable value for the voxel volume prevents computational redundancy caused by over-subdivision.

[0034] Furthermore, the step S3 includes the following steps:

[0035] S31. Calculate the texture consistency of the occluded area based on the gray-level co-occurrence matrix :

[0036]

[0037] in is the joint probability distribution of adjacent pixels in the gray-level co-occurrence matrix; the closer the texture consistency TC value is to 1, the more uniform the texture is and the closer it is to static occlusion; the closer the value is to 0, the more chaotic the texture is and the closer it is to dynamic occlusion;

[0038] S32, based on the texture consistency of the occluded area And point cloud density Build a texture-density matching model:

[0039]

[0040] Indicates the strength of the correlation between texture features and voxel point cloud density, , When it is >0, it indicates positive correlation, and the texture features and point cloud density change in the same direction. When <0, it indicates negative correlation, and the texture features and point cloud density change in the opposite direction. Time is not relevant;

[0041] For texture consistency And point cloud density The covariance of is used to measure the degree of joint change between the two:

[0042]

[0043] Where N is the number of voxels in the occluded area, is the standard deviation of texture consistency, is the standard deviation of the point cloud density;

[0044] S33, when Greater than threshold , it is judged as static occlusion.

[0045] Furthermore, the step S3 further includes the following steps:

[0046] S34, when When it approaches 0 or is negatively correlated, the time change rate of the occluded area is calculated using the time series data of the aerial image. :

[0047]

[0048] in, represents the pixel value of the occluded area in the t-th frame image, where T is the total number of frames;

[0049] S35, according to the time change rate of the occlusion area , texture consistency and density differences Calculate the dynamic index of the occluded area :

[0050]

[0051] when Greater than the dynamic threshold It is considered dynamic occlusion.

[0052] Further, the S4 includes the following steps:

[0053] S41. According to the three-dimensional occlusion rate index , quantized occlusion classification :

[0054]

[0055] Occlusion Grading The occlusion level is divided into low level, medium level and high level in the interval;

[0056] S42, when the occlusion type is static occlusion, if the occlusion is classified as low, it is marked as repairable; if the occlusion is classified as medium, it is marked as repairable according to texture consistency. Determine whether it is repairable. If the occlusion level is high, verify the geometric consistency of the occluded area to determine whether it is repairable. If it is repairable, repair the occluded area using one or more of the GAN image completion method, BIM model fusion, and point cloud interpolation algorithms.

[0057] S43. When the occlusion type is dynamic occlusion, if the occlusion classification is high, it is marked as unrepairable. If the occlusion classification is low or medium, it is judged whether it is repairable based on the time change rate TVR. If it is repairable, it is repaired using the optical flow method, time series completion method, and motion trajectory prediction algorithm.

[0058] Quantitative grading is performed using the three-dimensional occlusion rate indicator. For low-level occlusions, such as trees blocking facades, when the occluded area is small, the image can be directly completed through the GAN adversarial network and marked as repairable. For medium-level static occlusions, such as billboards blocking roofs, it is necessary to combine texture consistency to determine the possibility of repair. If there is sufficient reference data around the occluded area, such as point clouds and images of adjacent unoccluded areas, it is determined to be repairable. If the occluded area is isolated and there is no reference data, it is determined to be unrepairable. If it is a high-level static occlusion, such as a bridge blocking the ground, additional verification of the geometric continuity of the occluded area is required. If the occlusion type is dynamic occlusion, its time rate of change TVR can be used to determine whether temporal interpolation can be performed for repair.

[0059] The location of the unrepairable area can be determined by converting voxel coordinates into geographic coordinates. Each voxel has a unique index coordinate in the voxelized 3D grid, such as By pre-establishing a coordinate system conversion model, the voxel index coordinates are mapped to the geographic coordinate system. Specifically, when taking aerial photos with a drone, the GPS position, attitude angle, and camera parameters of each frame are synchronously recorded. Then, the collinearity equation in photogrammetry is used to establish a mapping relationship between the image pixel coordinates and the three-dimensional geographic coordinates. Combined with the resolution of the voxelized grid, the voxel index coordinates are converted to the corresponding geographic coordinates:

[0060]

[0061] Where T is the geographic coordinate origin is the spatial resolution of the voxel grid.

[0062] Then, for the set of voxels marked as unrepairable, a three-dimensional morphological closing operation is used to merge adjacent voxels to generate a continuous three-dimensional region. The boundary voxels of the merged region are extracted, and their geographic coordinate range is calculated. The minimum circumscribed cube is generated to describe the spatial extent of the unrepairable area. The boundary coordinates are stored in a three-dimensional point cloud format, and metadata such as timestamp, occlusion level, and occlusion type are recorded.

[0063] The present invention also discloses a digital twin oblique photography model construction system based on fine modeling. The above-mentioned digital twin oblique photography model construction method based on fine modeling is characterized by comprising a data acquisition module, an occlusion classification module, an occlusion type module, a repair judgment module and a path planning module;

[0064] The data acquisition module is used to take aerial photos using a drone equipped with a 5-lens oblique camera to obtain a 3D point cloud and aerial images of the shooting area;

[0065] The occlusion classification module is used to discretize the original 3D point cloud into voxels to obtain a voxelized 3D grid. Each voxel records the number of point clouds, calculates the point cloud density of each voxel based on the number of points within the voxel, and determines its 3D occlusion rate index based on the point cloud density of each voxel. Based on the 3D occlusion quantification index of each voxel block, it identifies the occluded area in the shooting area and determines the occlusion classification of the occluded area;

[0066] The occlusion type module is used to obtain a two-dimensional image of the occluded area through aerial images, map the two-dimensional image to the voxel coordinate system of the three-dimensional point cloud, associate the pixels in the two-dimensional image with the point cloud density in the voxels of the three-dimensional point cloud, calculate the correlation between the point cloud density of the pixels and the associated voxels at different time sequences based on the time series data, and determine the occlusion type;

[0067] A repair judgment module is used to identify whether the occlusion area is repairable based on the occlusion grade and occlusion type of the occlusion area. If it is repairable, a preset repair algorithm is selected based on the occlusion grade and occlusion type to repair the occlusion area. If it is unrepairable, the location of the unrepairable occlusion area is identified and stored;

[0068] The path planning module is used to generate the collection route of the ground mobile collection platform according to the location of all irreparable occlusion areas, and perform ground collection on the irreparable occlusion areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is a flow chart of an embodiment of the method for constructing a digital twin oblique photography model based on fine modeling of the present invention. DETAILED DESCRIPTION

[0070] The following is further described in detail through specific implementation methods:

[0071] The embodiment is basically as shown in the attached Figure 1 As shown:

[0072] The method for constructing a digital twin oblique photography model based on fine modeling includes the following steps:

[0073] S1. Use a drone equipped with a 5-lens oblique camera to take aerial photos and obtain a 3D point cloud and aerial images of the shooting area.

[0074] S2. Voxelize and discretize the original 3D point cloud to obtain a voxelized 3D grid. Each voxel records the number of point clouds. The point cloud density of each voxel is calculated based on the number of points within the voxel. The 3D occlusion rate index of each voxel is determined based on the point cloud density of each voxel. Based on the 3D occlusion quantification index of each voxel block, the occluded area in the captured area is identified and the occlusion classification of the occluded area is determined.

[0075] S3. Obtain a two-dimensional image of the occluded area using aerial imagery, map the two-dimensional image to a voxel coordinate system of a three-dimensional point cloud, associate pixels in the two-dimensional image with point cloud densities in voxels of the three-dimensional point cloud, calculate correlations between the point cloud densities of pixels and associated voxels at different time series based on the time series data, and determine the type of occlusion.

[0076] S4. Identify whether the occluded area is repairable based on the occlusion classification and occlusion type of the occluded area. If the occluded area is repairable, select a preset repair algorithm based on the occlusion classification and occlusion type to repair the occluded area. If the occluded area is unrepairable, identify the location of the unrepairable area and store it.

[0077] S5. Generate a collection route for the ground mobile collection platform based on the locations of all the irreparable occlusion areas, and perform ground collection on the irreparable occlusion areas.

[0078] The S2 comprises the following steps:

[0079] S21, inputting the original 3D point cloud data into an adaptive voxel partitioning algorithm, outputting a voxelized 3D grid, and recording the number of point clouds in each voxel;

[0080] S22. Calculate the point cloud density of each voxel:

[0081]

[0082] in is the voxel density, is the number of point clouds within the voxel, is the voxel volume;

[0083] S23. Calculate the density difference between the current voxel and the 26 neighboring voxels:

[0084]

[0085] in is the average density of the 26 neighborhoods;

[0086] S24. Calculate the visibility coefficient by ray casting algorithm , combined with point cloud density and density differences Calculate the three-dimensional occlusion rate index:

[0087]

[0088] in is the maximum reference value of point cloud density, according to the output three-dimensional occlusion rate index , when the three-dimensional occlusion rate index Above the preset occlusion threshold , the area where the voxel is located is marked as the occlusion area, and the three-dimensional occlusion rate index The higher it is, the higher the occlusion level.

[0089] The point cloud is voxelized and the original point cloud is input into the adaptive voxel partitioning algorithm to obtain a voxelized three-dimensional grid according to the preset voxel specifications, such as a 32*32*32 voxel grid, where each voxel is a cubic unit. Each voxel contains several point clouds. Based on the number of point clouds in the voxel, the point cloud density of the voxel is calculated. Since the occluded area usually forms a local "density depression", the density difference between the current voxel and the surrounding environment is measured through 26 fields. The 26 fields are 6 face fields, including the fields of 6 shared faces (up and down, front and back, left and right); the edge fields in the diagonal direction of 12 shared edges and the corner fields in the diagonal direction of 8 shared vertices, totaling 26 fields. If the density of a voxel is significantly lower than that of its surrounding environment, it is due to data loss caused by occlusion.

[0090] The S2 further comprises the following steps:

[0091] S25, identifying the number of abnormal voxels, when the number of abnormal voxels Below the preset quantity threshold And the voxel volume Shrink voxels larger than the preset voxel size lower limit When , adjust the voxel specifications and re-execute S21 to S24;

[0092] S26, when When , the current voxel division is retained and S3 is executed;

[0093] S27, when and When , it is determined that the occlusion is not significant and the adjustment is terminated;

[0094] The S26 and S27 ,in is the initial voxel size.

[0095] A minimum threshold for the number of abnormal voxels and a minimum allowable value for the voxel volume are preset. When the number of abnormal voxels falls below the minimum threshold, dual threshold constraints are used to distinguish between "low-density occlusion" and "non-occluded areas" by using the voxel specification lower limit and the number threshold. Dynamic adjustment strategies are applied to occlusion characteristics in different scenarios, improving algorithm robustness. Setting a minimum allowable value for the voxel volume prevents computational redundancy caused by over-subdivision.

[0096] The S3 includes the following steps:

[0097] S31. Calculate the texture consistency of the occluded area based on the gray-level co-occurrence matrix :

[0098]

[0099] in is the joint probability distribution of adjacent pixels in the gray-level co-occurrence matrix; the closer the texture consistency TC value is to 1, the more uniform the texture is and the closer it is to static occlusion; the closer the value is to 0, the more chaotic the texture is and the closer it is to dynamic occlusion;

[0100] S32, based on the texture consistency of the occluded area And point cloud density Build a texture-density matching model:

[0101]

[0102] Indicates the strength of the correlation between texture features and voxel point cloud density, , When it is >0, it indicates positive correlation, and the texture features and point cloud density change in the same direction. When <0, it indicates negative correlation, and the texture features and point cloud density change in the opposite direction. Time is not relevant;

[0103] For texture consistency And point cloud density The covariance of is used to measure the degree of joint change between the two:

[0104]

[0105] Where N is the number of voxels in the occluded area, is the standard deviation of texture consistency, is the standard deviation of the point cloud density;

[0106] S33, when Greater than threshold , it is judged as static occlusion.

[0107] The S3 further comprises the following steps:

[0108] S34, when When it approaches 0 or is negatively correlated, the time change rate of the occluded area is calculated using the time series data of the aerial image. :

[0109]

[0110] in, represents the pixel value of the occluded area in the t-th frame image, where T is the total number of frames;

[0111] S35, according to the time change rate of the occlusion area , texture consistency and density differences Calculate the dynamic index of the occluded area :

[0112]

[0113] when Greater than the dynamic threshold It is considered dynamic occlusion.

[0114] The S4 comprises the following steps:

[0115] S41, according to the three-dimensional occlusion rate index , quantized occlusion classification :

[0116]

[0117] Occlusion Grading The occlusion level is divided into low level, medium level and high level in the interval;

[0118] S42, when the occlusion type is static occlusion, if the occlusion is classified as low, it is marked as repairable; if the occlusion is classified as medium, it is marked as repairable according to texture consistency. Determine whether it is repairable. If the occlusion level is high, verify the geometric consistency of the occluded area to determine whether it is repairable. If it is repairable, repair the occluded area using one or more of the GAN image completion method, BIM model fusion, and point cloud interpolation algorithms.

[0119] S43. When the occlusion type is dynamic occlusion, if the occlusion classification is high, it is marked as unrepairable. If the occlusion classification is low or medium, it is judged whether it is repairable based on the time change rate TVR. If it is repairable, it is repaired using the optical flow method, time series completion method, and motion trajectory prediction algorithm.

[0120] In this embodiment, the location of the unrepairable area is determined by converting voxel coordinates into geographic coordinates. Each voxel has a unique index coordinate in the voxelized three-dimensional grid, such as By pre-establishing a coordinate system conversion model, the voxel index coordinates are mapped to the geographic coordinate system. Specifically, when taking aerial photos with a drone, the GPS position, attitude angle, and camera parameters of each frame are synchronously recorded. Then, the collinearity equation in photogrammetry is used to establish a mapping relationship between the image pixel coordinates and the three-dimensional geographic coordinates. Combined with the resolution of the voxelized grid, the voxel index coordinates are converted to the corresponding geographic coordinates:

[0121]

[0122] Where T is the geographic coordinate origin is the spatial resolution of the voxel grid.

[0123] Then, for the set of voxels marked as unrepairable, a three-dimensional morphological closing operation is used to merge adjacent voxels to generate a continuous three-dimensional region. The boundary voxels of the merged region are extracted, and their geographic coordinate range is calculated. The minimum circumscribed cube is generated to describe the spatial extent of the unrepairable area. The boundary coordinates are stored in a three-dimensional point cloud format, and metadata such as timestamp, occlusion level, and occlusion type are recorded.

[0124] The present invention uses a three-dimensional occlusion rate index for quantitative grading. For low-level occlusions, such as trees blocking facades, when the occluded area is small, the image can be directly completed through the GAN adversarial network and directly marked as repairable. For medium-level static occlusions, such as billboards blocking roofs, it is necessary to combine texture consistency to determine the possibility of repair. If there is sufficient reference data around the occluded area, such as point clouds and images of adjacent unoccluded areas, it is determined to be repairable. If the occluded area is isolated and there is no reference data, it is determined to be unrepairable. If it is a high-level static occlusion, such as a bridge blocking the ground, it is necessary to additionally verify the geometric continuity of the occluded area. If the occlusion type is dynamic occlusion, its time rate of change TVR can be used to determine whether time series interpolation can be performed for repair.

[0125] Using drones equipped with oblique photography equipment for aerial photography, 3D point clouds and aerial images of various urban areas are collected. The collected raw 3D point clouds are discretized into voxels to produce a voxelized 3D grid, with each voxel containing a number of point clouds. The point cloud density of each voxel is determined by analyzing the number of point clouds within the voxel. If occlusion occurs, the point cloud density within the occluded area will be significantly lower than that of other nearby voxels. Therefore, by identifying the point cloud density within the voxel and voxel clusters with abnormal point cloud density, the occluded area can be determined. Aerial images are then used to obtain a 2D image of the occluded area. By correlating the pixels in the 2D image with the point cloud density of the voxels in the 3D point cloud, the occlusion type can be identified. Occlusion types include static and dynamic. Dynamic occlusion refers to the movement of obstructing objects over time or space, resulting in dynamic changes in the position and shape of the obstructed area. Examples include pedestrians blocking buildings, vehicles blocking roads, and birds blocking drone footage. Static occlusion refers to an obstruction with a fixed position and shape, and the obstructed area remains consistent across multiple shots. For example, trees obscuring facades, billboards obscuring roofs, and bridges obscuring the ground. Dynamic occlusion manifests itself in imagery as temporal changes in pixel values, and in point clouds as abnormal fluctuations in point cloud density within voxels. Static occlusion manifests itself as a fixed texture in imagery and as stable, low-density voxels in point clouds. Therefore, by correlating the two, we can identify whether the obstructed area is static or dynamic.

[0126] After identifying the occlusion type and occlusion classification, the system determines whether the occlusion is repairable based on the occlusion classification and type. If it is repairable, the system uses a pre-set repair algorithm to repair the occlusion based on the occlusion classification and type. If it is unrepairable, the location of the unrepairable occlusion is stored and recorded, and a collection route for the ground mobile collection platform is generated based on the existing location of the unrepairable occlusion area to perform ground-based re-shooting of the unrepairable occlusion area.

[0127] Compared with the existing technology, in this application, the occlusion classification and occlusion type of the occlusion area are identified. According to the occlusion classification and occlusion type of the occlusion area, it is determined whether the occlusion area can be self-repaired through algorithms such as GAN adversarial network and calling BIM data completion. If it can be self-repaired, the corresponding algorithm can be called for automatic repair. If it cannot be self-repaired, the occluded area is re-photographed by ground re-shooting. Traditional manual labeling takes 5-8 hours for one square kilometer, and it takes a lot of time to label all areas. Therefore, it is usually necessary to re-shoot the occluded part of an area after marking it, and then mark the next area at the same time. The technical solution in this application can find all the occlusion areas that need to be re-photographed at one time, and plan the optimal re-shooting path according to the location of each occlusion area. Compared with the existing technology, a lot of time is saved in both area labeling and re-shooting planning, so it can be applied to the dynamic update of real-time urban models.

[0128] By correlating the voxel point cloud density in a 3D point cloud with the pixel values in a 2D image, occlusion types can be identified more accurately than with traditional object detection. For example, traditional object detection uses the outline of an object to determine the type of occlusion, potentially misidentifying a human-shaped sculpture as a pedestrian and thus identifying it as dynamic occlusion. However, this application considers only pixel values and point cloud density to determine occlusion type, enabling more precise determinations and providing accurate data for subsequent processing and analysis.

[0129] The present invention also discloses a digital twin oblique photography model construction system based on fine modeling. The above-mentioned digital twin oblique photography model construction method based on fine modeling is characterized by comprising a data acquisition module, an occlusion classification module, an occlusion type module, a repair judgment module and a path planning module;

[0130] The data acquisition module is used to take aerial photos using a drone equipped with a 5-lens oblique camera to obtain a 3D point cloud and aerial images of the shooting area;

[0131] The occlusion classification module is used to discretize the original 3D point cloud into voxels to obtain a voxelized 3D grid. Each voxel records the number of point clouds, calculates the point cloud density of each voxel based on the number of points within the voxel, and determines its 3D occlusion rate index based on the point cloud density of each voxel. Based on the 3D occlusion quantification index of each voxel block, it identifies the occluded area in the shooting area and determines the occlusion classification of the occluded area;

[0132] The occlusion type module is used to obtain a two-dimensional image of the occluded area through aerial images, map the two-dimensional image to the voxel coordinate system of the three-dimensional point cloud, associate the pixels in the two-dimensional image with the point cloud density in the voxels of the three-dimensional point cloud, calculate the correlation between the point cloud density of the pixels and the associated voxels at different time sequences based on the time series data, and determine the occlusion type;

[0133] A repair judgment module is used to identify whether the occlusion area is repairable based on the occlusion grade and occlusion type of the occlusion area. If it is repairable, a preset repair algorithm is selected based on the occlusion grade and occlusion type to repair the occlusion area. If it is unrepairable, the location of the unrepairable occlusion area is identified and stored;

[0134] The path planning module is used to generate the collection route of the ground mobile collection platform according to the location of all irreparable occlusion areas, and perform ground collection on the irreparable occlusion areas.

[0135] The above are only embodiments of the present invention. Common knowledge such as the known specific structures and characteristics in the scheme are not described in detail here. Ordinary technicians in the field are aware of all common technical knowledge in the technical field of the invention before the application date or priority date, can obtain all existing technologies in the field, and have the ability to apply conventional experimental means before that date. Ordinary technicians in the field can improve and implement this scheme in combination with their own abilities under the inspiration given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several variations and improvements can be made, which should also be regarded as the scope of protection of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A method for constructing a digital twin oblique photography model based on fine modeling, characterized by: The following steps are involved: S1. Use a drone equipped with a 5-lens oblique camera to take aerial photos and obtain a 3D point cloud and aerial images of the shooting area. S2. Voxelize and discretize the original 3D point cloud to obtain a voxelized 3D grid. Each voxel records the number of point clouds. The point cloud density of each voxel is calculated based on the number of points within the voxel. The 3D occlusion rate index of each voxel is determined based on the point cloud density of each voxel. Based on the 3D occlusion quantification index of each voxel block, the occluded area in the captured area is identified and the occlusion classification of the occluded area is determined. S3. Obtain a two-dimensional image of the occluded area using aerial imagery, map the two-dimensional image to a voxel coordinate system of a three-dimensional point cloud, associate pixels in the two-dimensional image with point cloud densities in voxels of the three-dimensional point cloud, calculate correlations between the point cloud densities of pixels and associated voxels at different time series based on the time series data, and determine the type of occlusion. S4. Identify whether the occluded area is repairable based on the occlusion classification and occlusion type of the occluded area. If the occluded area is repairable, select a preset repair algorithm based on the occlusion classification and occlusion type to repair the occluded area. If the occluded area is unrepairable, identify the location of the unrepairable area and store it. S5. Generate a collection route for the ground mobile collection platform based on the locations of all the irreparable occlusion areas, and perform ground collection on the irreparable occlusion areas.

2. The method for constructing a digital twin oblique photography model based on fine modeling according to claim 1, characterized in that: The S2 comprises the following steps: S21, inputting the original 3D point cloud data into an adaptive voxel partitioning algorithm, outputting a voxelized 3D grid, and recording the number of point clouds in each voxel; S22. Calculate the point cloud density of each voxel: in is the voxel density, is the number of point clouds within the voxel, is the voxel volume; S23. Calculate the density difference between the current voxel and the 26 neighboring voxels: in is the average density of the 26 neighborhoods; S24. Calculate the visibility coefficient by ray casting algorithm , combined with point cloud density and density differences Calculate the three-dimensional occlusion rate index: in is the maximum reference value of point cloud density, according to the output three-dimensional occlusion rate index , when the three-dimensional occlusion rate index Above the preset occlusion threshold , mark the voxel as an abnormal voxel, determine the occlusion area based on the abnormal voxel set, and the three-dimensional occlusion rate index The higher it is, the higher the occlusion level.

3. The method for constructing a digital twin oblique photography model based on fine modeling according to claim 2, characterized in that: The S2 further comprises the following steps: S25, identifying the number of abnormal voxels, when the number of abnormal voxels Below the preset quantity threshold And the voxel volume Shrink voxels larger than the preset voxel size lower limit When , adjust the voxel specifications and re-execute S21 to S24; S26, when When , the current voxel division is retained and S3 is executed; S27, when and When , it is determined that the occlusion is not significant and the adjustment is terminated; The S26 and S27 ,in is the initial voxel size.

4. The method for constructing a digital twin oblique photography model based on fine modeling according to claim 2, characterized in that: The S3 includes the following steps: S31. Calculate the texture consistency of the occluded area based on the gray-level co-occurrence matrix : in is the joint probability distribution of adjacent pixels in the gray-level co-occurrence matrix; the closer the texture consistency TC value is to 1, the more uniform the texture is and the closer it is to static occlusion; the closer the value is to 0, the more chaotic the texture is and the closer it is to dynamic occlusion; S32, based on the texture consistency of the occluded area And point cloud density Build a texture-density matching model: Indicates the strength of the correlation between texture features and voxel point cloud density, , When it is >0, it indicates positive correlation, and the texture features and point cloud density change in the same direction. When <0, it indicates negative correlation, and the texture features and point cloud density change in the opposite direction. Time is not relevant; For texture consistency And point cloud density The covariance of is used to measure the degree of joint change between the two: Where N is the number of voxels in the occluded area, is the standard deviation of texture consistency, is the standard deviation of the point cloud density; S33, when Greater than threshold , it is judged as static occlusion.

5. The method for constructing a digital twin oblique photography model based on fine modeling according to claim 4 is characterized in that: The S3 further comprises the following steps: S34, when When it approaches 0 or is negatively correlated, the time change rate of the occluded area is calculated using the time series data of the aerial image. : in, represents the pixel value of the occluded area in the t-th frame image, where T is the total number of frames; S35, according to the time change rate of the occlusion area , texture consistency and density differences Calculate the dynamic index of the occluded area : when Greater than the dynamic threshold It is considered dynamic occlusion.

6. The method for constructing a digital twin oblique photography model based on fine modeling according to claim 5, characterized in that: The S4 comprises the following steps: S41, according to the three-dimensional occlusion rate index , quantized occlusion classification : Occlusion Grading The occlusion level is divided into low level, medium level and high level in the interval; S42, when the occlusion type is static occlusion, if the occlusion is classified as low, it is marked as repairable; if the occlusion is classified as medium, it is marked as repairable according to texture consistency. Determine whether it is repairable. If the occlusion level is high, verify the geometric consistency of the occluded area to determine whether it is repairable. If it is repairable, repair the occluded area using one or more of the GAN image completion method, BIM model fusion, and point cloud interpolation algorithms. S43. When the occlusion type is dynamic occlusion, if the occlusion classification is high, it is marked as unrepairable. If the occlusion classification is low or medium, it is judged whether it is repairable based on the time change rate TVR. If it is repairable, it is repaired using the optical flow method, time series completion method, and motion trajectory prediction algorithm.

7. A digital twin oblique photography model construction system based on fine modeling, using the digital twin oblique photography model construction method based on fine modeling according to any one of claims 1 to 6, characterized in that: It includes data acquisition module, occlusion classification module, occlusion type module, repair judgment module and path planning module; The data acquisition module is used to take aerial photos using a drone equipped with a 5-lens oblique camera to obtain a 3D point cloud and aerial images of the shooting area; The occlusion classification module is used to discretize the original 3D point cloud into voxels to obtain a voxelized 3D grid. Each voxel records the number of point clouds, calculates the point cloud density of each voxel based on the number of points within the voxel, and determines its 3D occlusion rate index based on the point cloud density of each voxel. Based on the 3D occlusion quantification index of each voxel block, it identifies the occluded area in the shooting area and determines the occlusion classification of the occluded area; The occlusion type module is used to obtain a two-dimensional image of the occluded area through aerial images, map the two-dimensional image to the voxel coordinate system of the three-dimensional point cloud, associate the pixels in the two-dimensional image with the point cloud density in the voxels of the three-dimensional point cloud, calculate the correlation between the point cloud density of the pixels and the associated voxels at different time sequences based on the time series data, and determine the occlusion type; A repair judgment module is used to identify whether the occlusion area is repairable based on the occlusion grade and occlusion type of the occlusion area. If it is repairable, a preset repair algorithm is selected based on the occlusion grade and occlusion type to repair the occlusion area. If it is unrepairable, the location of the unrepairable occlusion area is identified and stored; The path planning module is used to generate the collection route of the ground mobile collection platform according to the location of all irreparable occlusion areas, and perform ground collection on the irreparable occlusion areas.

Images