A Digital Twin Vegetation Environment Modeling Method Based on Remote Sensing Images and CGA Rules

Abstract

The present invention discloses a digital twin vegetation environment modeling method based on remote sensing images and CGA rules, belonging to the technical field of remote sensing monitoring. It includes: S1: the image data preprocessing stage; S2: the target detection model training and optimization stage; S3: the non-dense vegetation single target detection and recognition stage, using the trained deep neural network model to perform single tree target detection and recognition on the ortho-panchromatic image of the target area, obtaining the recognition detection frame, and performing vector data conversion to obtain geographic vector data; S4: the modeling environment preprocessing stage, performing environment preprocessing on the modeling environment to obtain the basic modeling scene; S5: the model creation and export stage, based on the geographic vector data and the basic modeling scene, creating a single vegetation model to obtain the vegetation environment model scene. It bypasses the surveying and mapping means with high hardware, software, and labor costs such as laser point cloud and oblique photography to restore and model the vegetation environment, and has the advantages of strong feasibility and low comprehensive cost.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing monitoring, and particularly to a digital twin vegetation environment modeling method based on remote sensing images and CGA rules. Background Art

[0002] As a top-level enabling technology system for digital new infrastructure, the digital twin technology is widely used in scenarios such as infrastructure construction and operation digitization, industrial production digitization, smart cities, and smart transportation. As a part of the scenario construction process, vegetation modeling generally uses surveying and mapping technologies such as drone aerial photography to collect oblique photography or laser point clouds for reconstructing vegetation real-scene models. Its advantages are that the technology is mature and can largely ensure the authenticity and accuracy of model restoration. However, its disadvantages such as high computing and storage costs, poor data compatibility, poor near-view rendering effects, and poor linkage are all reasons restricting the further development and application of such environmental modeling technologies in digital twin scenarios. The vegetation models constructed by 3D modeling software such as 3DSMAX and MAYA are more intuitive and realistic, but most 3D modeling software lacks data communication interfaces with geographic information system (GIS) software, including domestic SuperMap platforms and international mainstream 3D GIS platforms such as ArcGIS Pro. Eventually, it is very difficult to quickly deploy the created real-scene models to local or cloud-based three-dimensional geographic information system scenarios (3DGIS) for basic calculations or rendering. At the same time, its modeling for large-scale scenarios is time-consuming and laborious, and the cost becomes uncontrollable, and it is more suitable for vegetation modeling in small-scale digital twin scenarios. Summary of the Invention

[0003] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a digital twin vegetation environment modeling method based on remote sensing images and CGA rules.

[0004] The purpose of the present invention is achieved by the following technical solutions: A digital twin vegetation environment modeling method based on remote sensing images and CGA rules, comprising the following steps:

[0005] S1: In the image data preprocessing stage, satellite image data is obtained, including ortho-panchromatic image data and digital elevation model data; the satellite image data is preprocessed to obtain first satellite image data; the first satellite image data is subjected to target detection mask calibration to obtain training data and export it;

[0006] S2: In the target detection model training and optimization stage, a deep neural network model is constructed and trained using the training data to obtain a trained deep neural network model;

[0007] S3: In the non-dense vegetation single-object detection and recognition stage, use the trained deep neural network model to detect and recognize single-tree objects in the ortho-panchromatic image of the target area, obtain the recognition detection bounding boxes, and perform vector data conversion to obtain geographic vector data;

[0008] S4: In the modeling environment preprocessing stage, build the modeling environment and perform environmental preprocessing on the modeling environment to obtain the basic modeling scene;

[0009] S5: In the model creation and export stage, based on the geographic vector data and the basic modeling scene, create a single-vegetation model to obtain the vegetation environment model scene.

[0010] Preferably, in the S1: Image data preprocessing stage, the following steps are further included:

[0011] S11: Select the data source and obtain satellite image data from the data source;

[0012] S12: Import the satellite image data into the map drawing and analysis software; use the mosaicking tool to perform mosaicking operations on the satellite image data, and use the coordinate transformation method to resample the satellite image data, thereby transforming it into the first satellite image data based on the standard Gaussian projection coordinate system;

[0013] S13: Load the first satellite image data into the map workspace of the map drawing and analysis software and make the target data set in the selected state; then perform mask plotting on the target data set, and perform rectangular bounding box plotting on the ortho-panchromatic image data to obtain the training data.

[0014] Preferably, in the S2: Target detection model training and tuning stage, the following steps are further included:

[0015] S21: Use the cross-entropy loss function and the regression loss function to judge the training loss of the deep neural network model, and use the SGD optimizer combined with momentum and weight decay to optimize the training process of the deep neural network model;

[0016] S22: Adjust the hyperparameters to reach the first preset quantization index; test the deep neural network model, and perform training tuning according to the test results until the deep neural network model reaches the second preset quantization index to obtain the trained deep neural network model.

[0017] Preferably, in the S3: Non-dense vegetation single-object detection and recognition stage, the following steps are further included:

[0018] S31: Use the raster spatial data conversion library to perform sliding sampling on the ortho-panchromatic image of the target area to obtain the sampled image data, and the confidence level B of detecting tree plants during the sliding sampling TreeGreater than or equal to the preset confidence level; the sampling sliding step length of the sliding sampling is less than the number of pixels of the sampling sample, and the sampling sliding step length includes a horizontal sliding step length S X and a vertical sliding step length S Y Let the size of the sampling sample be X sample ×Y sample pixels, 0 < horizontal sliding step length S X < X sample 0 < vertical sliding step length S Y < Y sample ;

[0019] S32: Input the sampled image data into the trained deep neural network model for data prediction to obtain multiple recognition detection bounding boxes; the center coordinates P m ( x m , y m ) are calculated through the affine transformation matrix T of the ortho-panchromatic image of the target area and the pixel coordinate space relationship between the sliding window of the sliding sampling and the ortho-panchromatic image of the target area. The calculation formula is as follows:

[0020]

[0021] In the formula, x m is the east distance value of P m , in meters, and satisfies x m ≥ 0; y m is the north distance value of P m , in meters, and satisfies y m ≥ 0; x is the pixel abscissa value of the upper left pixel point of the current sampled image data located in the ortho-panchromatic image of the target area, in pixels, x ≥ 0; y is the pixel ordinate value of the upper left pixel point of the current sampled image data located in the ortho-panchromatic image of the target area, in pixels, and satisfies y ≥ 0; x 1 is the pixel abscissa of the leftmost side of the tree plant detection result window with a confidence level B Tree > the preset confidence level within the range of the sampled image data, and satisfies x 1 ≥ 0, in pixels; x 2 is the pixel abscissa of the rightmost side of the tree plant detection result window with a confidence level B Tree > the preset confidence level within the range of the sampled image data, and satisfies x 2 ≥ 0, in pixels;y 1 is the confidence level B Tre e > the vertical coordinate of the pixel at the topmost position within the sampling image data range of the tree plant detection result form with a preset confidence level, and satisfies y 1 ≥ 0, in pixels; y 2 is the confidence level B Tree > the vertical coordinate of the pixel at the bottommost position within the sampling image data range of the tree plant detection result form with a preset confidence level, and satisfies y 2 ≥ 0, in pixels; T 11 is the first element in the first row of the affine transformation matrix T, whose meaning is the northing of the top - left anchor pixel of the ortho - panchromatic image of the target area in the corresponding projection coordinate system, and satisfies T 11 ≥ 0, in meters; T 12 is the second element in the first row of the affine transformation matrix, whose meaning is the width of the horizontal pixels of the ortho - panchromatic image of the target area, and satisfies 12 > 0, in meters; T 13 is the third element in the first row of the affine transformation matrix, whose meaning is the horizontal pixel rotation angle of the ortho - panchromatic image of the target area; T 21 is the first element in the last row of the affine transformation matrix, whose meaning is the easting of the top - left anchor pixel of the ortho - panchromatic image of the target area in the corresponding projection coordinate system, and satisfies T 21 ≥ 0, in meters; T 22 is the second element in the last row of the affine transformation matrix, whose meaning is the vertical pixel rotation angle of the ortho - panchromatic image of the target area; T 23 is the third element in the last row of the affine transformation matrix, whose meaning is the width of the vertical pixels of the ortho - panchromatic image of the target area, and satisfies 23 < 0, in meters;

[0022] The area of the recognition detection frame S Box Is calculated by the following formula:

[0023] ;

[0024] Store the center coordinates P m ( x m , y m ) and the area S Box in a CSV file, and x m 、 y m 、 S BoxStored separately as three independent fields and processed by the data table processing function in the mapping analysis software x m and y m are converted into two-dimensional coordinate points, and finally vector data conversion is performed to obtain geographic vector data.

[0025] Preferably, in the S4: modeling environment preprocessing stage, the following steps are further included:

[0026] S41: Use digital elevation model data for digital terrain modeling, and the data range of the digital elevation model data is the same as that of the ortho-panchromatic image data; use the ArcPy library to vectorize the digital elevation model data, output the contour lines in the digital elevation model data as polyline vectors, and then perform line smoothing processing on the polyline vectors to obtain input vector line features; then create a Triangulated Irregular Network (TIN), extract three-dimensional surface features containing elevation information from the TIN and export them as three-dimensional surface feature data in a preset format;

[0027] S42: Perform interpolation operations on the geographic vector data to obtain a three-dimensional point set containing elevation information;

[0028] S43: Create a constant raster; import the three-dimensional surface feature data and the constant raster into the three-dimensional modeling software; import the three-dimensional point set into the three-dimensional modeling software as the data source for calculating the generation positions of individual vegetation models;

[0029] S44: Use the three-dimensional surface feature data to match the height values of the constant raster layer to obtain a basic modeling scene.

[0030] Preferably, in the S5: model creation and export stage, the following steps are further included:

[0031] S51: Define the main rules, attribute declarations, and parameter interaction panels; use the three-dimensional point set as the scope of the main rules, traverse all selected three-dimensional points using the CGA rules, and then initialize the annotation rules of the three-dimensional points and execute random instance insertion, instance scaling, and random rotation one by one, and perform random angle rotation on the vegetation around the vertical axis and scaling factors in any direction for each individual vegetation model instance f Calculated by the following formula;

[0032]

[0033] In the formula, S Mean is the cross-sectional area in the horizontal direction of the outer envelope of the current individual vegetation model; T MAX is the upper limit of the allowable floating range; T MINis the lower limit of the allowable floating range; N r is Gaussian white noise, which conforms to the distribution characteristics with a mean of 0 and a variance of 1;

[0034] Declare the panel group in the parameter interaction panel, and then f Declare the attributes of the parameters in the calculation formula of the scaling factor;

[0035] S52: Select the data source for calculating the generation position of the single vegetation model, create the single vegetation model, and modify the visualization data;

[0036] S53: Export the single vegetation model after modifying the visualization data into a 3D model exchange format to obtain the vegetation environment model scene in the target data format.

[0037] Preferably, the deep neural network model is the Faster-RCNN model.

[0038] Preferably, the map drawing analysis software is ArcGIS Pro software.

[0039] The beneficial effects of the present invention are:

[0040] 1) The vegetation modeling in the non-densely vegetated area using open-source data bypasses the high-cost surveying methods such as laser point clouds and oblique photography in terms of hardware, software, and labor costs to restore and model the vegetation environment, and has the advantages of strong feasibility and relatively low comprehensive cost.

[0041] 2) The deep neural network model for target detection data recognition in the present invention is based on the relatively general programming framework of Pytorch and the model architecture of Faster-RCNN. It is relatively easy to model for the target detection task at the satellite image level of vegetation, has good model efficiency, and strong transferability.

[0042] 3) Through the optimization at the terrain modeling level, combined with the Bezier curve smoothing of contour lines and the irregular triangular network modeling, the 12.5-meter digital elevation model data source can further improve the resolution of the DEM after optimization.

[0043] 4) The non-densely vegetated modeling based on the CGA rule uses code to generate the model, which greatly reduces the modeling time and labor costs, and reserves an interactive adjustment interface, improving the adjustable freedom of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flowchart of a digital twin vegetation environment modeling method based on remote sensing images and CGA rules. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0045] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0046] Refer to Figure 1 , the present invention provides a technical solution: a digital twin vegetation environment modeling method based on remote sensing images and CGA rules, including the following steps:

[0047] S1: In the image data preprocessing stage, satellite image data is obtained, including orthophoto panchromatic image data and digital elevation model data; the satellite image data is preprocessed to obtain the first satellite image data; the target detection mask calibration is performed on the first satellite image data to obtain training data and export it.

[0048] S2: In the target detection model training and optimization stage, a deep neural network model is constructed and trained using the training data to obtain a trained deep neural network model.

[0049] S3: In the non-dense vegetation monomer target detection and recognition stage, the trained deep neural network model is used to detect and recognize monomer tree targets in the orthophoto panchromatic image of the target area, obtain the recognition detection frame, and perform vector data conversion to obtain geographic vector data.

[0050] S4: In the modeling environment preprocessing stage, a modeling environment is built, and the environment preprocessing is performed on the modeling environment to obtain a basic modeling scene.

[0051] S5: In the model creation and export stage, based on the geographic vector data and the basic modeling scene, a monomer vegetation model is created to obtain a vegetation environment model scene.

[0052] In this embodiment, in S1, first, open-source high-resolution satellite image data is obtained, including ortho-panchromatic images and digital elevation models (DEMs); then, data preprocessing is performed on them, and then training data is exported, including target detection mask calibration and training data export; in S2, the Faster-RCNN architecture in the deep neural network is adopted. In the case of non-dense vegetation, that is, when the plants are sparse, this model is used to identify and locate individual vegetation. Note that this step of the model is applicable to ortho-panchromatic images in the visible light band, and when the resolution of the image data is relatively high, that is, when the horizontal resolution of the corresponding map image product reaches 0.25 m in each direction, the algorithm has better detection results for vegetation targets. This step includes model code writing, model training, and testing. In S3, vector data conversion is performed to ultimately enable coordinate positioning and related calculations based on the GIS system. In S4, a basic modeling environment is built to create conditions for vegetation modeling and improve the conformity of the built model with the real world. The geographic vector data in S3 is two-dimensional coordinates and does not have elevation attributes, so it cannot reflect the changing trend of vegetation with terrain undulation in the real situation. Therefore, it is first necessary to construct a three-dimensional terrain surface based on the ALOS open-source terrain data, and then use surface interpolation means to perform interpolation operations on the two-dimensional point set data to endow it with elevation attributes. Finally, after importing into CityEngine, a basic modeling scene is formed. Therefore, this process mainly includes terrain modeling, vector interpolation, data import, and environmental preprocessing. In S5, based on the geographic vector data in S3 and the basic modeling scene obtained from elevation modeling in S4, and using the CGA rule code of CityEngine, an individual vegetation model is created. The size of the individual vegetation model is related to the area of the recognition detection frame, and the tree species is related to the tree types in the area. Tree models generally use the model resources in the built-in library of CityEngine or third-party self-built model resources, and perform random transformations on the models based on the "s function" and "r function" to make the generated models have a certain degree of randomness and the visual effect of the results is more natural. The finally obtained model scene uses common model formats such as FBX and obj for data exchange, so that the non-dense vegetation environment in the real world can be restored in other general digital twin scenarios. Therefore, this step covers steps such as CGA rule writing, model generation and adjustment, and scene export.

[0053] In some embodiments, in the S1: image data preprocessing stage, the following steps are further included:

[0054] S11: Select a data source and obtain satellite image data from the data source;

[0055] S12: Import the satellite image data into map drawing and analysis software; use the mosaic tool to perform mosaic operations on the satellite image data, and use the coordinate conversion method to resample the satellite image data, so as to transform it into the first satellite image data based on the standard Gaussian projection coordinate system;

[0056] S13: Load the first satellite image data into the map workspace of the map drawing and analysis software and select the target data set; then perform mask mapping on the target data set and perform rectangular frame mapping on the orthophoto panchromatic image data to obtain training data.

[0057] In this embodiment, mainstream satellite image products such as Google Map or Bing Map are selected as the main data source of orthophoto panchromatic image data in S11, and the ALOS platform 12.5m resolution data source is selected as the main data source of digital elevation. The above-mentioned raster data is generally stored in GeoTIFF format with the suffix ".tif". In S12, the source data is imported through ArcGIS Pro software, and the scattered data sets that were originally multiple independent files are mosaicked using mosaic tools, and the original Google image and DEM data are resampled using coordinate transformation methods to transform them into data based on a standard Gaussian projection coordinate system that matches the target modeling range. In S13, general data processing tools for data label masks in the direction of machine vision in deep learning, such as LableAnything, LableMe, etc., can usually be used, while the present invention requires the use of ArcGIS Pro's own label annotation tool for mask annotation to obtain compatibility with image data containing geographic location information. First, load the orthophoto panchromatic image data into the ArcGIS Pro software map workspace and ensure that the target dataset to be processed is selected; secondly, use the "Mark objects for deep learning" tool in the classification toolbar to mask the dataset. For orthophoto full-color satellite images, they are mainly used for non-intensive vegetation, such as parks, urban traffic greening, and other single vegetation or tree crowns in the image that can be identified by the naked eye, so it is sufficient to mark them with a rectangular frame. The classification label category only has the category of single trees. The results are exported through the "Export dataset for deep learning" tool and the "metadata format" is guaranteed to be "RCNN", and finally the training and test / validation sets are formed.

[0058] In some embodiments, the target detection model training and tuning stage S2 further includes the following steps:

[0059] S21: Use the cross entropy loss function and regression loss function to determine the training loss of the deep neural network model, and use the SGD optimizer combined with momentum and weight decay to optimize the training process of the deep neural network model;

[0060] S22: Adjust the hyperparameters to achieve a first preset quantitative index; test the deep neural network model, and perform training and optimization according to the test results until the deep neural network model reaches a second preset quantitative index to obtain a trained deep neural network model.

[0061] In this embodiment, in S21, the deep learning programming framework PyTorch based on the Python programming language is used for code writing. The pre-trained ResNet-50 model is used as the backbone model for convolutional neural network image classification, and the Faster-RCNN algorithm architecture is mainly used for object detection. It should be noted that this approach is mainly to perform deep learning tasks without relying on the ArcGIS Pro software environment, improving the improvement space and scalability of the subsequent deep neural network model. Secondly, it is necessary to define the loss function and optimizer in the deep neural network model. In this solution, the stable cross-entropy loss and regression loss are respectively used as the loss definition indicators for the two tasks of classification and positioning window, and the sum of the two is used as the overall training loss determination indicator for the deep neural network model. Finally, the SGD optimizer is used in combination with momentum and weight decay to optimize the training process of the deep neural network model. In S22, it first involves the training and tuning process of the deep neural network model. This process requires tuning the hyperparameters to make its performance meet the corresponding quantitative indicators. Secondly, it is the testing of the deep neural network model. This process tests the results of the trained model in combination with the quantitative indicators. The training and tuning process of the deep neural network model mainly focuses on whether there are situations such as gradient disappearance and gradient explosion in the loss iteration, and dynamically tunes in combination with non-descriptive model structure parameters such as the learning rate, batch size, and number of iterations. Finally, the model is improved until the quantitative indicators of its final performance evaluation meet the requirements. For example, the F1 score of the deep neural network model is greater than 0.75. Relevant single or multiple quantitative indicators are freely selected by technicians and are not limited in this invention. For the finally trained data, the model is stored in the data format ONNX of the general deep learning model to ensure its portability and generality. The testing of the deep neural network model results is a key step in verifying the performance of the model using the reserved validation set data. Generally, 10% of the randomly selected and separated data from the original training set is used for the final test. And the final test index results of the deep neural network model of this invention meet the same trend as the relevant performance indicators obtained at the end of the training of the deep neural network model. For example, the F1 score of the deep neural network model meets the requirement of being greater than 0.75, otherwise it represents that the overall performance of the model is poor.

[0062] In some embodiments, in step S3: the non-dense vegetation single object detection and recognition stage, the following steps are further included:

[0063] S31: Use the raster spatial data conversion library to perform sliding sampling on the ortho-panchromatic image of the target area to obtain sampled image data, and the confidence level B of detecting tree plants during the sliding sampling Tree is greater than or equal to the pre-set confidence level; the sampling sliding step of the sliding sampling is less than the number of pixels of the sampling sample, and the sampling sliding step includes the horizontal sliding step S Xand the vertical sliding step S Y , let the size of the sampling sample be X sample ×Y sample pixels, 0 < the horizontal sliding step S X < X sample 、0 < the vertical sliding step S Y < Y sample ;

[0064] S32: Input the sampled image data into the trained deep neural network model for data prediction to obtain multiple recognition detection bounding boxes; the center coordinates P m ( x m , y m ) are calculated through the affine transformation matrix T of the ortho-panchromatic image of the target area, the spatial relationship between the sliding window of sliding sampling and the pixel coordinates of the ortho-panchromatic image of the target area, and its calculation formula is as follows:

[0065]

[0066] In the formula, x m is the east distance value of P m , in meters, and satisfies x m ≥0; y m is the north distance value of P m , in meters, and satisfies y m ≥0; x is the pixel abscissa value of the upper left pixel point of the current sampled image data located in the ortho-panchromatic image of the target area, in pixels, x ≥0; y is the pixel ordinate value of the upper left pixel point of the current sampled image data located in the ortho-panchromatic image of the target area, in pixels, and satisfies y ≥0; x 1 is the pixel abscissa of the leftmost side of the tree plant detection result window with a confidence B Tree > the preset confidence within the range of the sampled image data, and satisfies x 1≥0, in pixels; x 2 is the pixel abscissa of the rightmost side of the tree plant detection result window with a confidence B Tree > the preset confidence within the range of the sampled image data, and satisfies x 2≥0, in pixels; y 1 is the pixel ordinate of the topmost side of the tree plant detection result window with a confidence B Tre e> the preset confidence within the range of the sampled image data, and satisfiesy 1 ≥ 0, with the unit being pixels; y 2 is the confidence level B Tree > The vertical coordinate of the pixel at the bottommost position within the sampling image data range of the tree plant detection result form with a confidence level greater than the preset confidence level, and it satisfies y 2 ≥ 0, with the unit being pixels; T 11 is the first element in the first row of the affine transformation matrix T, and its meaning is that the pixel at the top - leftmost anchor point of the ortho - panchromatic image of the target area is located at the north distance in the corresponding projection coordinate system, and it satisfies T 11 ≥ 0, with the unit being meters; T 12 is the second element in the first row of the affine transformation matrix, and its meaning is the width of the horizontal pixels of the ortho - panchromatic image of the target area, and it satisfies T 12 > 0, with the unit being meters; T 13 is the third element in the first row of the affine transformation matrix, and its meaning is the horizontal pixel rotation angle of the ortho - panchromatic image of the target area; T 21 is the first element in the last row of the affine transformation matrix, and its meaning is that the pixel at the top - leftmost anchor point of the ortho - panchromatic image of the target area is located at the east distance in the corresponding projection coordinate system, and it satisfies T 21 ≥ 0, with the unit being meters; T 22 is the second element in the last row of the affine transformation matrix, and its meaning is the vertical pixel rotation angle of the ortho - panchromatic image of the target area; T 23 is the third element in the last row of the affine transformation matrix, and its meaning is the width of the vertical pixels of the ortho - panchromatic image of the target area, and it satisfies T 23 < 0, with the unit being meters;

[0067] The area of the recognition detection frame S Box It is calculated by the following formula:

[0068] ;

[0069] The center coordinates P of the recognition detection frame m ( x m , y m ) and the area S Box are stored in a CSV file, and x m 、 y m 、 S Box are stored as three independent fields respectively and are processed by the data table processing function in the map drawing analysis software to make x m 、 y mConvert it into two-dimensional coordinate points, and finally perform vector data conversion to obtain geographic vector data.

[0070] In this embodiment, S31 calls the trained deep neural network model to predict remote sensing data. The "torch.load" method is used to load the results of the trained deep neural network model, and the data is preprocessed based on the GDAL library of Python. Usually, the geotransformation and geographic coordinate system of the remote sensing data need to be obtained in advance to ensure that the geographic information of the image is not affected. Note that the large-scale remote sensing observation data is usually large. Therefore, a mode of batch prediction and then stitching of the data in the form of data segmentation or sliding window is required. Otherwise, in most cases, it will be affected by computer hardware limitations and the efficiency of the model in calculating excessive data, resulting in the inability to normally run the recognition calculation of vegetation single targets. The present invention adopts an image sliding sampling mode to improve the calculation efficiency. First, the implementer can customize a function based on the GDAL library to perform small-range sliding sampling from the original image, and note that the sampling sliding step size is often smaller than the number of pixels in the data acquisition range. For example, if the sampling sample size is X sample ×Y sample pixels, then the horizontal and vertical sliding step sizes S X 、S Y must satisfy 0 < S X < X sample and 0 < S Y < Y sample . The confidence B Tree of the detected tree plant results satisfies B Tree ≥0.75, that is, the probability that the model determines this target to be a tree plant is greater than or equal to 75%. In S32, the sampled image data is used as the input end, and the loaded deep neural network model is used to perform data prediction, and the target forms of several prediction results are obtained. The reaction in the results is the pixel horizontal and vertical coordinate values of the four corner points of the corresponding form in the image. Based on GDAL, the affine transformation matrix T of the original input ortho-panchromatic image is obtained, as well as the pixel coordinate space relationship between the sliding window and the original image, so that the center coordinate P m ( x m , y m ) of a certain recognition detection frame can be deduced. In its calculation formula, T 13 、T 22 generally takes a value of 0, and the unit is decimal degrees. By calculating the center coordinate P m ( x m , y m ) of each predicted recognition detection frame, the four corner point coordinates of each predicted recognition detection frame can be obtained, and the area S Box, which is used to evaluate the size of trees. The predicted result coordinates and the corresponding recognized detection box areas are stored in a CSV file. The X coordinate of the point, the Y coordinate of the point, and the area of the corresponding recognized detection box are stored as three independent fields Point_X, Point_Y, and S_Box in the file, separated by English commas. Then, using the data table processing function "XYTableToPoint" in the geographic data processing library ArcPy based on Python, it is converted from the data table to two-dimensional coordinate points. In the conversion parameters, {x_field} and {y_field} correspond to the Point_X and Point_Y fields in the aforementioned data table respectively, and the rest of the parameters except the input and output file paths are kept with default values empty. Finally, it is converted into a geographic vector data in the shapefile format with the suffix ".shp" using "FeatureClassToShapefile".

[0071] In some embodiments, in the S4: modeling environment preprocessing stage, the following steps are further included:

[0072] S41: Use digital elevation model data to perform digital terrain modeling. The data range of the digital elevation model data is the same as that of the ortho-panchromatic image data; Use the ArcPy library to vectorize the digital elevation model data, output the contour lines in the digital elevation model data as polyline vectors, and then perform line smoothing processing on the polyline vectors to obtain the input vector line features; Then create a Triangulated Irregular Network (TIN), extract the three-dimensional surface features containing elevation information from the TIN and export them as three-dimensional surface feature data in a preset format;

[0073] S42: Perform interpolation operations on the geographic vector data to obtain a three-dimensional point set containing elevation information;

[0074] S43: Create a constant raster; Import the three-dimensional surface feature data and the constant raster into the 3D modeling software; Import the three-dimensional point set into the 3D modeling software as the data source for calculating the generation positions of the individual vegetation models;

[0075] S44: Use the three-dimensional surface feature data to perform height value matching on the constant raster layer to obtain the basic modeling scene.

[0076] In this embodiment, S41 is to construct a three-dimensional digital terrain of the terrain through a grid or topographic map, and a digital elevation model (DEM) approximate to the real ground needs to be obtained. First, the present invention models the digital terrain based on the open-source DEM data source of ALOS, and the obtained data range should be consistent with the range of the ortho-panchromatic image initially selected by the present invention. Implementers can crop and adjust according to the downloaded data, and its coordinate system should be transformed with the coordinate parameters corresponding to the processed area, such as the Gauss projection coordinate system based on Datum 2000. Secondly, vectorize the DEM data based on the ArcPy library, use the "Contour" function to process the contour lines of the raster data, and output them as polyline vectors in the ".shp" format. The {contour_interval} parameter is a required item, and the smaller the value, the smaller the elevation interval between adjacent contour lines. Smooth the polyline vectors based on the above output using the "cartography.SmoothLine" function of ArcPy. The {algorithm} parameter is recommended to select "BEZIER_INTERPOLATION", that is, the method of fitting the Bezier curve between the break points to obtain smoother contour lines. The {error_option} parameter is recommended to select "RESOLVE_ERRORS" to solve the geometric topology errors of internal intersection or self-intersection of line features. Further, use the smoothed contour lines as the input vector line features, and use the "ddd.CreateTin" function to create an irregular triangular network (TIN). The {in_features_1, height_field_1, SF_type_1,...} in the parameters it receives mainly involve the first vector data source (which can be multiple) and its participating height fields and constraint types, and need to be set corresponding to the first input contour line vector data source, the element corresponding height value field, and the linear constraint type is a soft constraint. Finally, use the "ddd.TinTriangle" function with the above TIN data as the data input source to extract three-dimensional surface elements containing elevation information, and output the data in the shapefile format through the data export function "FeatureClassToShapefile". (Note: In the present invention, the expressions in the form of "{...}" all refer to the input parameter names of the functions in the programming library) S42 is to obtain elevation values for the two-dimensional plane point data through the three-dimensional terrain, so that the vegetation conforms to the real spatial position distribution. Use the irregular triangular network and the predicted vegetation coordinate point set shapefile file as the input data sources of the first parameter {in_surface} and the second parameter {in_feature_class} in turn, and use the "ddd.InterpolateShape" function in ArcPy to perform interpolation operations on the vegetation coordinate point set, and output a three-dimensional point set containing elevation information.First in S43, use the "CreateConstantRaster" function in ArcPy to create a constant raster. The parameter {constant_value} is the input constant value, usually set to 0; {data_type} corresponds to the output data type, usually set to the "FLOAT" floating-point value; {cell_size} corresponds to the output cell size, and the balance between output accuracy and data volume should be considered. Generally, the value range is 0.5 - 2; {extent} is the output range, which should be the same as the output vector triangulated terrain range. After running the process, a floating-point raster with all cell values of 0 and in the GeoTIFF format is obtained. Subsequently, import the 3D surface feature file in the shapefile format and the constant raster in the GeoTIFF format into CityEngine software. The constant raster is imported using the Terrain import mode. In the input parameters, use the full-color orthophoto image at the input end during the prediction of the deep neural network model as the texture data source in the import parameters to obtain a better display effect of the terrain model. Finally, import the 3D point set data as the data source for calculating the generation position of the single-vegetation model. In S44, the imported vectors (including 3D surface features and 3D point sets) and raster data layers are used as the data basis for constructing the scene model, and data preprocessing still needs to be performed to form a complete modeling basic environment. After selecting the 3D surface feature layer, use the "Align terrain to shapes" command to match the height values of the constant raster layer using the 3D surface features to obtain a digital elevation model (DEM) that conforms to the terrain undulation. In particular, in CityEngine 2022.1 and earlier versions, after the previously imported constant raster is selected, the minimum cell resolution can be adjusted as needed. Its value defaults to 1024. The larger the value, the finer the resolution, the richer the details that can be obtained from the matched target 3D surface features, and at the same time, the corresponding increase in the hardware resources consumed by the operation.

[0077] In some embodiments, step S5: model creation and export stage further includes the following steps:

[0078] S51: Define the main rule, property declaration, and parameter interaction panel; use the 3D point set as the main rule scope, traverse all selected 3D points using the CGA rule, then initialize the annotation rules of the 3D points and execute random instance insertion, instance scaling, and random rotation one by one. Rotate the vegetation randomly around the vertical axis and scale factors in any direction for each single-vegetation model instance f Calculate through the following formula;

[0079]

[0080] In the formula, S Meanis the cross-sectional area in the horizontal direction of the peripheral envelope of the current single-vegetation model; T MAX is the upper limit of the allowable floating range; T MIN is the lower limit of the allowable floating range; N r is Gaussian white noise, which conforms to the distribution characteristics with a mean of 0 and a variance of 1;

[0081] Declare the panel group in the parameter interaction panel, and then declare the attributes of the parameters in the calculation formula of the scaling factor f ;

[0082] S52: Select the data source for calculating the generation position of the single-vegetation model, create the single-vegetation model, and modify the visualization data;

[0083] S53: Export the single-vegetation model with modified visualization data as a three-dimensional model exchange format to obtain the vegetation environment model scene in the target data format.

[0084] In this embodiment, S51 first writes the CGA rules, which are mainly divided into three parts: the main rule, attributes, and parameter interaction panel. The main rule specifies the parent node name corresponding to the starting rule, which is generally named StartRule or Lot, and its definition ends with "-->"; the main rule is subsequently connected to any geometric element operation or calculation function and subsequent derived child nodes, and its calculations are executed in sequence. In this solution, based on the imported three-dimensional point elements as the main rule scope, the CGA rules will traverse all the selected three-dimensional points, initialize their main rules, and execute subsequent geometric operations one by one. For subsequent geometric operations, this solution adopts three steps in sequence: random instance insertion + instance scaling + random rotation. The instance insertion function is "i(file_path)", and its main function is to insert an external 3D model with the point initialized by the main rule as the base point, where "file_path" is the file path of the tree model in the resource library, and it can also be the file path of other third-party model files; the instance scaling function is "s(scale_x, scale_z, scale_y)", and its main function is to scale the size based on the outer envelope of the inserted model. scale_x, scale_z, and scale_y are the scaling factors in the horizontal transverse, horizontal longitudinal, and vertical directions respectively, and their values are floating-point numbers; the instance rotation function is "r(angle_x,angle_z, angle_y)", and its main function is to rotate the angle based on the model insertion base point. angle_x, angle_z, and angle_y are the rotation angles around the horizontal transverse axis, horizontal longitudinal axis, and vertical axis passing through the base point respectively. In this solution, only the vegetation is randomly rotated by an angle around the vertical axis, and the random angle can be generated by the "rand" function. It should be noted that when scaling the size of the vegetation, the "S_Box" field carried in the aforementioned input point set should be considered, that is, the area of the recognition detection box S Box , calculate the scaling factors in each direction of the size of each vegetation model instance f After the scaling transformation and random rotation based on the area dimension, the model will be more natural. Attribute declaration is a predefined process for the relevant parameters manipulated in the rule part of the CGA code. The present invention incorporates the parameters in the calculation formula of the scaling factor f into the consideration of attribute predefined. Among them, the "file_path" input parameter of the "i function" is the file path, and the value type is a string. Therefore, multiple model paths can be predefined as attributes and placed in an array as elements for random call; T MAX and T MIN are directly assigned initial values as needed when static attributes are predefined; S Box and SMean It is a dynamic attribute with a predefined initial value of 0. When the data needs to be called, the field name can be used as the right value and the attribute name as the left value for assignment through the equal sign. The parameter interaction panel provides a space for non-coded parameter interaction interface customization expansion for modelers in the "Inspector" parameter indicator of the software. Before the attribute declaration in the CGA code, it is necessary to supplement with "@Group(group_name)" for panel group declaration, where the "group_name" parameter is the group name, and it is combined in the form of "@Order(order_sequence)" + "interaction type". The former declaration and the "order_sequence" parameter specify the position of the attribute interaction field, and the latter specifies the type of attribute interaction. Then, follow the declaration of the corresponding attribute for initialization. After the CGA code in S52 is written, select all the three-dimensional point sets and use the "Generate models of selected shapes" instruction in CityEngine to automatically generate the model to obtain a preliminary single-vegetation model. The implementer observes and modifies the visualization data of a specific vegetation model based on the interaction panel of the "Inspector" parameter indicator, including its scaling ratio, rotation parameters, vegetation type path, etc. In S53, export all the results to a common three-dimensional model exchange format, such as ".obj", ".FBX", etc. After selecting the generated and adjusted single-vegetation model and the DEM layer in the "Scene" layer display window, use the "ExportModels" instruction to output the selected model and obtain the final scene model result in the target data format, that is, the vegetation environment model scene in the target data format.

[0085] In some embodiments, the deep neural network model is a Faster-RCNN model.

[0086] In some embodiments, the map drawing analysis software is ArcGIS Pro software.

[0087] The present invention uses ortho-color satellite images in the visible light mode based on open source and open source digital elevation data sources with a resolution of 12.5 meters as the data basis for non-intensive vegetation recognition and positioning. Compared with the ortho-aerial photography acquisition mode with higher data resolution, the technical path of the present invention has a higher degree of freedom because the data covers almost the whole world, and the implementer can select any area to obtain data without having to collect data through on-site aerial photography; based on the above key points, the standardized label and training data annotation processing flow in the ArcGIS Pro desktop geographic information system platform is adopted for deep learning training data annotation and export. Compared with other deep learning label annotation and processing processes and tools, the present invention is more applicable to tasks centered on geographical location information.

[0088] For the situation of non-intensive vegetation coverage, the present invention uses the Faster-RCNN architecture with a relatively balanced balance between the difficulty of construction and the performance of the model as the modeling basis, which can bring good recognition accuracy even when the resolution of the ortho-color image data is slightly inferior to that of the on-site aerial photography data; and based on the ONNX general deep learning model format, it brings a certain cross-platform migration ability for the vegetation recognition model, so that technicians do not have to repeatedly build a task model for non-intensive vegetation monomer recognition.

[0089] The present invention selects GDAL and pytorch to jointly call the output results of the recognition model, and successively performs sliding sampling target recognition of the input data map sheet and calculation of the center coordinates of the detection window based on the affine transformation matrix, so that the frame of vegetation recognition can be seamlessly output as a geographical coordinate table centered on the target monitoring result frame while carrying geographical information, and can also bring out the corresponding detection frame area as field information, obtaining the occupied area value of the detection target, laying a foundation for the subsequent estimation of the model size.

[0090] Determining the spatial coordinates of the terrain and vegetation basic models through terrain modeling and vegetation coordinate vector interpolation helps to restore the real vegetation distribution. The contour smoothing based on Bessel curves and the terrain surface reconstruction of triangular meshes improve the quality of DEM data, thereby improving the accuracy of the spatial distribution of the non-intensive vegetation scene model involved in the present invention.

[0091] Automatically generating a model for the vegetation model based on the CGA rule, performing model size and direction rotation transformation on the model based on the recognition window area, and supplementing with random white noise to make the generated vegetation model closer to the natural state; at the same time, an interface for later manual adjustment of model parameters is reserved, so that the present invention can improve a certain degree of freedom while taking into account efficiency.

[0092] The above are only the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be changed within the scope of the concept described herein through the above teachings or the technology or knowledge in the relevant field. And any changes and variations made by those skilled in the art without departing from the spirit and scope of the present invention shall fall within the protection scope of the appended claims of the present invention.

Claims

1. A digital twin vegetation environment modeling method based on remote sensing images and CGA rules, characterized in that: Including the following steps: S1: Image data preprocessing stage. Obtain satellite image data, including ortho-panchromatic image data and digital elevation model data; perform data preprocessing on the satellite image data to obtain the first satellite image data; perform target detection mask calibration on the first satellite image data to obtain training data and export it; S2: Target detection model training and optimization stage. Construct a deep neural network model and use the training data for training to obtain a trained deep neural network model; S3: Non-dense vegetation single-object detection and recognition stage. Use the trained deep neural network model to perform single-tree object detection and recognition on the ortho-panchromatic image of the target area, obtain the recognition detection bounding box, and perform vector data conversion to obtain geographic vector data; S4: Modeling environment preprocessing stage. Set up the modeling environment and perform environment preprocessing on the modeling environment to obtain the basic modeling scene; S5: Model creation and export stage. Based on the geographic vector data and the basic modeling scene, create a single-vegetation model to obtain the vegetation environment model scene; The S5: Model creation and export stage further includes the following steps: S51: Define the main rules, property declarations, and parameter interaction panel; use the three-dimensional point set as the scope of the main rules, traverse all selected three-dimensional points using CGA rules, then initialize the annotation rules for the three-dimensional points and execute random instance insertion, instance scaling, and random rotation one by one. Randomly rotate the vegetation around the vertical axis and scale factors in any direction for each individual vegetation model instance are calculated by the following formula; f Calculated by the following formula; Wherein, S Box is the area for identifying the detection frame body; S Mean is the cross-sectional area in the horizontal direction of the peripheral envelope of the current single vegetation model; T MAX is the upper limit of the allowable floating range; T MIN is the lower limit of the allowable floating range; N r is Gaussian white noise, which conforms to the distribution characteristics with a mean of 0 and a variance of 1; Declare a panel group in the parameter interaction panel, and then f Declare the attributes of the parameters in the calculation formula; S52: Select the data source for calculating the generation position of the single-vegetation model, create the single-vegetation model, and perform visualization data modification; S53: Export the single-vegetation model after visualization data modification to a 3D model exchange format to obtain the vegetation environment model scene in the target data format.

2. The digital twin vegetation environment modeling method based on remote sensing images and CGA rules according to claim 1, wherein: The S1: Image data preprocessing stage further includes the following steps: S11: Select the data source and obtain satellite image data from the data source; S12: Import the satellite image data into the map drawing and analysis software; use the mosaicking tool to perform mosaicking operations on the satellite image data, and use the coordinate transformation method to resample the satellite image data, thereby transforming it into the first satellite image data based on the standard Gaussian projection coordinate system; S13: Load the first satellite image data into the map workspace of the map drawing and analysis software and make the target data set in the selected state; then perform mask plotting on the target data set, and perform rectangular bounding box plotting on the ortho-panchromatic image data to obtain the training data.

3. The digital twin vegetation environment modeling method based on remote sensing images and CGA rules according to claim 1, wherein: The S2: Target detection model training and optimization stage further includes the following steps: S21: Use the cross-entropy loss function and the regression loss function to judge the training loss of the deep neural network model, and use the SGD optimizer combined with momentum and weight decay to optimize the training process of the deep neural network model; S22: Adjust the hyperparameters to reach the first preset quantization index; test the deep neural network model, and perform training optimization according to the test results until the deep neural network model reaches the second preset quantization index to obtain a trained deep neural network model.

4. The digital twin vegetation environment modeling method based on remote sensing images and CGA rules according to claim 1, characterized in that: The S3: Non-dense vegetation single-object detection and recognition stage further includes the following steps: S31: Use the raster space data conversion library to perform sliding sampling on the ortho-panchromatic image of the target area to obtain sampled image data, and the confidence level B of the tree plants detected by the sliding sampling Tree is greater than or equal to the preset confidence level; the sampling sliding step of the sliding sampling is less than the number of pixels of the sampling sample, and the sampling sliding step includes a horizontal sliding step S X and a vertical sliding step S Y . Let the size of the sampling sample be X sample ×Y sample pixels, 0 < horizontal sliding step S X < X sample , 0 < vertical sliding step S Y < Y sample ; S32: Input the sampled image data into the trained deep neural network model for data prediction to obtain multiple recognition detection bounding boxes; the center coordinates P of the recognition detection bounding boxes m ( x m , y m ) are calculated through the affine transformation matrix T of the ortho-panchromatic image of the target area, the spatial relationship between the sliding window of the sliding sampling and the pixel coordinates of the ortho-panchromatic image of the target area, and its calculation formula is as follows: Wherein, x m is the east distance value of P m in meters and satisfies x m ≥ 0; y m is the north distance value of P m in meters and satisfies y m ≥ 0; x is the pixel abscissa value of the upper left pixel of the current sampled image data located in the ortho-panchromatic image of the target area, in pixels, x ≥ 0; y is the pixel ordinate value of the upper left pixel of the current sampled image data located in the ortho-panchromatic image of the target area, in pixels and satisfies y ≥ 0; x 1 is the confidence level B Tree > the pixel abscissa of the leftmost side of the tree plant detection result window with a confidence level B greater than the preset confidence level within the sampled image data range and satisfies x 1 ≥ 0, in pixels; x 2 is the confidence level B Tree > the pixel abscissa of the rightmost side of the tree plant detection result window with a confidence level B greater than the preset confidence level within the sampled image data range and satisfies x 2 ≥ 0, in pixels; y 1 is the confidence level B Tre e > the pixel ordinate of the topmost side of the tree plant detection result window with a confidence level B greater than the preset confidence level within the sampled image data range and satisfies y 1 ≥ 0, in pixels; y 2 is the confidence level B Tree > the pixel ordinate of the bottommost side of the tree plant detection result window with a confidence level B greater than the preset confidence level within the sampled image data range and satisfies y 2 ≥ 0, in pixels; T 11 is the first element in the first row of the affine transformation matrix T, whose meaning is the north distance of the top left anchor pixel of the ortho-panchromatic image of the target area in the corresponding projection coordinate system and satisfies T 11 ≥ 0, in meters; T 12 is the second element in the first row of the affine transformation matrix, whose meaning is the width of the horizontal pixels of the ortho-panchromatic image of the target area and satisfies T 12 > 0, in meters; T 13 is the third element in the first row of the affine transformation matrix, whose meaning is the horizontal pixel rotation angle of the ortho-panchromatic image of the target area; T 21 is the first element in the last row of the affine transformation matrix, whose meaning is the east distance of the top left anchor pixel of the ortho-panchromatic image of the target area in the corresponding projection coordinate system and satisfies T 21 ≥0, in meters; T 22 is the second element in the last row of the affine transformation matrix, and its meaning is the longitudinal rotation angle of the pixels of the ortho-panchromatic image of the target area; T 23 is the third element in the last row of the affine transformation matrix, and its meaning is the width of the longitudinal pixels of the ortho-panchromatic image of the target area, and it satisfies T 23 <0, in meters; Identify the area of the detection frame S Box Calculate using the following formula: ； The center coordinate P of the detection frame is identified m ( x m , y m ),area S Box Stored in CSV files, and x m , y m , S Box They are stored as three independent fields and converted into x m , y m Convert it into two-dimensional coordinate points, and finally convert the vector data to obtain geographic vector data.

5. The digital twin vegetation environment modeling method based on remote sensing images and CGA rules according to claim 4, characterized in that: The S4: Modeling environment preprocessing stage further includes the following steps: S41: Use digital elevation model data for digital terrain modeling, where the data range of the digital elevation model data is the same as that of the ortho-panchromatic image data; use the ArcPy library to vectorize the digital elevation model data, output the contour lines in the digital elevation model data as polyline vectors, and then perform line smoothing on the polyline vectors to obtain the input vector line features; then create a Triangulated Irregular Network (TIN), extract the three-dimensional surface features containing elevation information from the TIN and export them as three-dimensional surface feature data in a preset format; S42: Perform interpolation operations on the geographic vector data to obtain a three-dimensional point set containing elevation information; S43: Create a constant raster; import the three-dimensional surface feature data and the constant raster into 3D modeling software; import the three-dimensional point set into the 3D modeling software as the data source for calculating the generation positions of individual vegetation models; S44: Use the three-dimensional surface feature data to match the height values of the constant raster layer to obtain the basic modeling scene.

6. The digital twin vegetation environment modeling method based on remote sensing images and CGA rules according to any one of claims 1-5, characterized in that: The depth neural network model is the Faster-RCNN model.

7. The method for digital twin vegetation environment modeling based on remote sensing images and CGA rules according to claim 2 or 4, characterized in that: The map drawing and analysis software is ArcGIS Pro software.

Images