Digital twinborn vegetation environment modeling method based on remote sensing image and CGA rule

Abstract

The invention discloses a digital twinborn vegetation environment modeling method based on a remote sensing image and a CGA rule, and belongs to the technical field of remote sensing monitoring. Comprising the following steps: S1, an image data preprocessing stage; s2, a target detection model training and tuning stage; s3, a non-dense vegetation monomer target detection and recognition stage: performing monomer tree target detection and recognition on the orthographic panchromatic image of the target area by using the trained deep neural network model to obtain a recognition and detection frame, and performing vector data conversion to obtain geographic vector data; s4, a modeling environment preprocessing stage: performing environment preprocessing on the modeling environment to obtain a basic modeling scene; and S5, a model creation and export stage: based on the geographic vector data and the basic modeling scene, creating a single vegetation model to obtain a vegetation environment model scene. The vegetation environment is restored and modeled by bypassing surveying and mapping means with relatively high hardware, software and labor cost, such as laser point cloud and oblique photography, and the method has the advantages of high feasibility and relatively low comprehensive cost.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing monitoring technology, and in particular to a digital twin vegetation environment modeling method based on remote sensing images and CGA rules. Background Art

[0002] As the top-level enabling technology system of digital new infrastructure, digital twin technology is widely used in infrastructure construction and digital operation and maintenance, digital industrial production, smart cities, smart transportation and other scenarios. As a part of the scene construction process, vegetation modeling generally uses drone aerial photography to collect oblique photography or laser point cloud and other surveying and mapping technologies to reconstruct vegetation real-scene models. Its advantages are mature technology and can largely guarantee the authenticity and accuracy of model restoration. However, its disadvantages such as high computing and storage costs, poor data compatibility, poor close-range rendering effects, and poor linkage are all reasons that restrict the further development and application of such environmental modeling technologies in digital twin scenarios. The vegetation model constructed by 3D modeling software such as 3DSMAX and MAYA is more intuitive and realistic, but most 3D modeling software lacks data communication interfaces for geographic information system (GIS) software, including the domestic SuperMap platform and the internationally commonly used ArcGIS Pro and other mainstream 3D GIS platforms. Ultimately, it is difficult to quickly deploy the created real-scene model to a local or cloud-based 3D geographic information system scene (3DGIS) for basic calculations or rendering. At the same time, its modeling under large-scale scene conditions is time-consuming and labor-intensive, and the cost becomes uncontrollable, so it is more suitable for small-scale digital twin scene vegetation modeling. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a digital twin vegetation environment modeling method based on remote sensing images and CGA rules.

[0004] The objective of the present invention is achieved through the following technical solutions: A digital twin vegetation environment modeling method based on remote sensing images and CGA rules, comprising the following steps: S1: Image data preprocessing stage, obtaining satellite image data, including orthophoto panchromatic image data and digital elevation model data; performing data preprocessing on the satellite image data to obtain first satellite image data; performing target detection mask calibration on the first satellite image data to obtain training data and export it; S2: Target detection model training and tuning phase, building a deep neural network model and using training data to train it to obtain a trained deep neural network model; S3: In the stage of detecting and identifying single tree targets in non-dense vegetation, the trained deep neural network model is used to detect and identify single tree targets in the orthophoto panchromatic image of the target area, obtain the detection frame, and convert the vector data into geographic vector data; S4: Modeling environment preprocessing stage, building the modeling environment and performing environmental preprocessing on the modeling environment to obtain the basic modeling scene; S5: Model creation and export stage, based on geographic vector data and basic modeling scenarios, a single vegetation model is created to obtain a vegetation environment model scene.

[0005] Preferably, the S1: image data preprocessing stage further includes the following steps: S11: Select a data source and obtain satellite image data from the data source; S12: importing the satellite image data into the map drawing and analysis software; using the mosaicking tool to perform mosaicking operation on the satellite image data, and using the coordinate conversion method to resample the satellite image data, thereby transforming it into the first satellite image data based on the standard Gauss projection coordinate system; S13: Load the first satellite image data into the map workspace of the map drawing and analysis software and select the target data set; then perform mask mapping on the target data set and perform rectangular frame mapping on the orthophoto panchromatic image data to obtain training data.

[0006] Preferably, the target detection model training and tuning stage S2 further includes the following steps: S21: Use the cross entropy loss function and regression loss function to determine the training loss of the deep neural network model, and use the SGD optimizer combined with momentum and weight decay to optimize the training process of the deep neural network model; S22: Adjust the hyperparameters to achieve a first preset quantitative index; test the deep neural network model, and perform training and optimization according to the test results until the deep neural network model reaches a second preset quantitative index to obtain a trained deep neural network model.

[0007] Preferably, the S3: non-dense vegetation single-unit target detection and identification stage further includes the following steps: S31: Use the raster spatial data conversion library to slide sample the orthophoto panchromatic image of the target area to obtain sampled image data, and the sliding sampling detects the confidence level of trees and plants B Tree Greater than or equal to the preset confidence; the sampling sliding step of the sliding sampling is less than the number of pixels of the sample, and the sampling sliding step includes the lateral sliding step S X and the longitudinal sliding step length S Y , let the size of the sample be X sample ×Y sample Pixels, 0<horizontal sliding step length S X <X sample , 0<longitudinal sliding step length S Y <Y sample ; S32: Input the sampled image data into the trained deep neural network model for data prediction to obtain multiple recognition detection frames; the center coordinates P of the recognition detection frames m ( x m , y m ) is calculated by the affine transformation matrix T of the orthophoto panchromatic image of the target area, the sliding window of the sliding sampling and the pixel coordinate space relationship of the orthophoto panchromatic image of the target area, and its calculation formula is as follows: In the formula, x m P m The easting value of is in meters and satisfies x m ≥0; y m P m The northing value of is in meters and satisfies y m ≥0; x The pixel horizontal coordinate value of the orthophoto panchromatic image of the target area where the upper left corner pixel of the current sampled image data is located, in pixels. x ≥0; y The pixel ordinate value of the orthophoto panchromatic image of the target area where the upper left corner pixel of the current sampled image data is located, in pixels, and satisfies y ≥0; x 1 is the confidence level B Tree >The tree plant detection result window with preset confidence is located at the leftmost pixel horizontal coordinate within the sampled image data range and meets x 1≥0, unit is pixel; x 2 is the confidence level B Tree >The tree plant detection result window with preset confidence is located at the rightmost pixel horizontal coordinate within the sampled image data range and meets x 2≥0, unit is pixel; y 1 is the confidence level B Tre e> The tree plant detection result window with preset confidence is located at the top pixel vertical coordinate within the sampling image data range and meets y 1≥0, unit is pixel; y 2 is the confidence level B Tree >The tree plant detection result window with preset confidence is located at the bottom pixel vertical coordinate within the sampled image data range and meets y 2≥0, unit is pixel; T 11is the first element of the first row in the affine transformation matrix T, which means that the upper left corner anchor pixel of the orthophoto panchromatic image of the target area is located at the north distance of the corresponding projection coordinate system and satisfies T 11 ≥0, in meters; T 12 is the second element in the first row of the affine transformation matrix, which means the width of the horizontal pixels of the orthophoto panchromatic image of the target area and satisfies T 12 >0, in meters; T 13 It is the third element in the first row of the affine transformation matrix, which means the horizontal rotation angle of the pixels of the orthophoto panchromatic image of the target area; T 21 is the first element of the last row in the affine transformation matrix, which means that the upper left corner anchor pixel of the orthophoto panchromatic image of the target area is located at the easting of the corresponding projection coordinate system and satisfies T 21 ≥0, in meters; T 22 It is the second element of the last row in the affine transformation matrix, which means the vertical rotation angle of the pixel of the orthophoto panchromatic image of the target area; T 23 is the third element of the last row in the affine transformation matrix, which means the width of the vertical pixels of the orthophoto panchromatic image of the target area and satisfies T 23 <0, unit is meter; Identify the area of ​​the detection frame S Box Calculated by the following formula: ; The center coordinate P of the detection frame is identified m ( x m , y m ),area S Box Stored in CSV files, and x m , y m , S Box They are stored as three independent fields and converted into x m , y m Convert it into two-dimensional coordinate points, and finally convert the vector data to obtain geographic vector data.

[0008] Preferably, the S4: modeling environment preprocessing stage further includes the following steps: S41: Perform digital terrain modeling using digital elevation model data, wherein the digital elevation model data has the same data range as the orthophoto panchromatic image data; perform vectorization processing on the digital elevation model data using the ArcPy library, output contour lines in the digital elevation model data as polyline vectors, and then perform line smoothing processing on the polyline vectors to obtain input vector line elements; then create an irregular triangulated network TIN, extract three-dimensional surface elements containing elevation information from the irregular triangulated network TIN, and export them as three-dimensional surface element data in a preset format; S42: performing an interpolation operation on the geographic vector data to obtain a three-dimensional point set containing elevation information; S43: creating a constant grid; importing the three-dimensional surface feature data and the constant grid into the three-dimensional modeling software; importing the three-dimensional point set into the three-dimensional modeling software as a data source for calculating the generation position of the single vegetation model; S44: Use the three-dimensional surface feature data to match the height value of the constant raster layer to obtain a basic modeling scene.

[0009] Preferably, the S5: model creation and export phase further includes the following steps: S51: Define the main rule, attribute declaration and parameter interaction panel; use the 3D point set as the main rule scope, use the CGA rule to traverse all selected 3D points, then initialize the 3D point injection rules and perform random instance insertion, instance scaling, and random rotation one by one, and rotate the vegetation around the vertical axis at a random angle. The scaling factor of each single vegetation model instance in any direction f Calculated by the following formula; In the formula, S Mean It is the horizontal cross-sectional area of ​​the outer envelope of the current single vegetation model; T MAX The upper limit of the allowable floating range; T MIN The lower limit of the allowable floating range; N r is Gaussian white noise, which has a distribution characteristic of mean 0 and variance 1; In the Parameter Interaction panel, declare the panel group, and then adjust the scaling factor f Parameters in the calculation formula are declared as attributes; S52: Selecting a data source for calculating a location for generating a single vegetation model, creating a single vegetation model, and performing visual data modification; S53: Exporting the single vegetation model after the visualization data modification into a three-dimensional model exchange format to obtain a vegetation environment model scene in a target data format.

[0010] Preferably, the deep neural network model is a Faster-RCNN model.

[0011] Preferably, the mapping analysis software is ArcGIS Pro software.

[0012] The beneficial effects of the present invention are: 1) Vegetation modeling in areas with non-dense vegetation coverage using open source data bypasses the use of surveying and mapping methods such as laser point cloud and oblique photography, which have high hardware, software, and labor costs, to restore and model the vegetation environment. It has the advantages of strong feasibility and low overall cost.

[0013] 2) The deep neural network model used for target detection data recognition in the present invention is based on the more common Pytorch programming framework and the Faster-RCNN model architecture. The target detection task modeling for vegetation satellite imagery is relatively easy to implement, with good model performance and strong transferability.

[0014] 3) Through optimization at the terrain modeling level, combined with Bezier curve smoothing of contour lines and irregular triangulated network modeling, the 12.5-meter digital elevation model data source can be optimized to further improve the resolution of the DEM.

[0015] 4) Non-dense vegetation modeling based on CGA rules. Using code to generate the model greatly reduces the time and labor cost of modeling, and reserves an interactive adjustment interface to improve the adjustable freedom of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of the digital twin vegetation environment modeling method based on remote sensing images and CGA rules. DETAILED DESCRIPTION

[0017] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0018] See also Figure 1 The present invention provides a technical solution: a digital twin vegetation environment modeling method based on remote sensing images and CGA rules, comprising the following steps: S1: Image data preprocessing stage, obtaining satellite image data, including orthophoto panchromatic image data and digital elevation model data; performing data preprocessing on the satellite image data to obtain first satellite image data; performing target detection mask calibration on the first satellite image data to obtain training data and export it; S2: Target detection model training and tuning phase, building a deep neural network model and using training data to train it to obtain a trained deep neural network model; S3: In the stage of detecting and identifying single tree targets in non-dense vegetation, the trained deep neural network model is used to detect and identify single tree targets in the orthophoto panchromatic image of the target area, obtain the detection frame, and convert the vector data into geographic vector data; S4: Modeling environment preprocessing stage, building the modeling environment and performing environmental preprocessing on the modeling environment to obtain the basic modeling scene; S5: Model creation and export stage, based on geographic vector data and basic modeling scenarios, a single vegetation model is created to obtain a vegetation environment model scene.

[0019] In this embodiment, S1 first obtains open-source high-resolution satellite image data, including orthophoto panchromatic images and digital elevation models (DEMs); then preprocesses the data, and then exports the training data, including target detection mask calibration and training data export; S2 uses the Faster-RCNN architecture in the deep neural network, and uses this model to identify and locate vegetation monomers in non-dense vegetation, that is, when the plants are sparse. Note that this step model is suitable for orthophoto panchromatic images in the visible light band, and when the image data resolution is high, that is, when the horizontal resolution of the corresponding map image product reaches 0.25m in each direction, the algorithm has better results for vegetation target detection. This step includes model code writing, model training and testing. The vector data conversion in S3 is to ultimately enable coordinate positioning and related calculations based on the GIS system. S4 creates conditions for vegetation modeling by building a basic modeling environment to improve the consistency of the built model with the real world. The geographic vector data in S3 is a two-dimensional coordinate, without elevation attributes, and cannot reflect the trend of vegetation changing with terrain in real situations. Therefore, it is necessary to first build a three-dimensional terrain surface based on the ALOS open source terrain data, and then use the surface interpolation method to interpolate the two-dimensional point set data to give it elevation attributes, and finally import it into CityEngine to form a basic modeling scene. Therefore, this process mainly includes terrain modeling, vector interpolation, data import, and environmental preprocessing. S5 is based on the geographic vector data in S3 and the basic modeling scene obtained by elevation modeling in S4, and uses CityEngine's CGA rule code to create a single vegetation model. The size of the vegetation single model is related to the area of ​​the identification detection frame, and the tree species is related to the tree type in the area. The tree model generally uses the model resources in the CityEngine built-in library or the third-party self-built model resources, and randomly transforms the model based on the "s function" and "r function" to make the generated model have a certain degree of randomness and the look and feel of the results is more natural. The final model scene uses common model formats such as FBX and obj for data exchange, so that the real-world non-intensive vegetation environment can be restored in other general digital twin scenes. Therefore, this step covers CGA rule writing, model generation and adjustment, scene export and other steps.

[0020] In some embodiments, the S1: image data preprocessing stage further includes the following steps: S11: Select a data source and obtain satellite image data from the data source; S12: importing the satellite image data into the map drawing and analysis software; using the mosaicking tool to perform mosaicking operation on the satellite image data, and using the coordinate conversion method to resample the satellite image data, thereby transforming it into the first satellite image data based on the standard Gauss projection coordinate system; S13: Load the first satellite image data into the map workspace of the map drawing and analysis software and select the target data set; then perform mask mapping on the target data set and perform rectangular frame mapping on the orthophoto panchromatic image data to obtain training data.

[0021] In this embodiment, mainstream satellite image products such as Google Map or Bing Map are selected as the main data source of orthophoto panchromatic image data in S11, and the ALOS platform 12.5m resolution data source is selected as the main data source of digital elevation. The above-mentioned raster data is generally stored in GeoTIFF format with the suffix ".tif". In S12, the source data is imported through ArcGIS Pro software, and the scattered data sets that were originally multiple independent files are mosaicked using mosaic tools, and the original Google image and DEM data are resampled using coordinate transformation methods to transform them into data based on a standard Gaussian projection coordinate system that matches the target modeling range. In S13, general data processing tools for data label masks in the direction of machine vision in deep learning, such as LableAnything, LableMe, etc., can usually be used, while the present invention requires the use of ArcGIS Pro's own label annotation tool for mask annotation to obtain compatibility with image data containing geographic location information. First, load the orthophoto panchromatic image data into the ArcGIS Pro software map workspace and ensure that the target dataset to be processed is selected; secondly, use the "Mark objects for deep learning" tool in the classification toolbar to mask the dataset. For orthophoto full-color satellite images, they are mainly used for non-intensive vegetation, such as parks, urban traffic greening, and other single vegetation or tree crowns in the image that can be identified by the naked eye, so it is sufficient to mark them with a rectangular frame. The classification label category only has the category of single trees. The results are exported through the "Export dataset for deep learning" tool and the "metadata format" is guaranteed to be "RCNN", and finally the training and test / validation sets are formed.

[0022] In some embodiments, the target detection model training and tuning stage S2 further includes the following steps: S21: Use the cross entropy loss function and regression loss function to determine the training loss of the deep neural network model, and use the SGD optimizer combined with momentum and weight decay to optimize the training process of the deep neural network model; S22: Adjust the hyperparameters to achieve a first preset quantitative index; test the deep neural network model, and perform training and optimization according to the test results until the deep neural network model reaches a second preset quantitative index to obtain a trained deep neural network model.

[0023] In this embodiment, S21 uses the deep learning programming framework PyTorch based on the Python programming language for code writing, and uses the pre-trained ResNet-50 model as the backbone model for convolutional neural network image classification. The target detection mainly uses the Faster-RCNN algorithm architecture. And it is noted that this approach is mainly to perform deep learning tasks independently of the ArcGIS Pro software environment, and to improve the improvement space and scalability of subsequent deep neural network models. Secondly, it is necessary to define the loss function and optimizer in the deep neural network model. This scheme uses stable cross entropy loss and regression loss as loss definition indicators for the two tasks of classification and positioning window, respectively, and the sum of the two is used as the training loss judgment indicator of the deep neural network model as a whole. Finally, the SGD optimizer is used to optimize the deep neural network model training process in combination with momentum and weight decay. S22 first involves the deep neural network model training tuning process, which requires tuning and controlling the hyperparameters to make their performance meet the corresponding quantitative indicators; secondly, the deep neural network model test, which tests the trained model results in combination with quantitative indicators. The training and tuning process of the deep neural network model mainly focuses on whether there are gradient vanishing and gradient exploding in the loss iteration, and dynamically tunes the non-descriptive model structure parameters such as learning rate, batch size, and number of iterations, and finally improves the model until the quantitative index of its final performance evaluation meets the requirements, such as the F1 score of the deep neural network model is greater than 0.75. The relevant single or multiple quantitative indicators are freely selected by the technicians, and the present invention does not limit it. The data of the final training is stored in the general deep learning model data format ONNX to ensure its portability and versatility. The result test of the deep neural network model is a key step in the performance verification of the model using the reserved validation set data. Generally, 10% of the random data is selected and separated in advance in the original training set for the final test, and the final test index result of the deep neural network model of the present invention meets the same trend as the relevant performance index obtained after the deep neural network model training, such as: the F1 score of the deep neural network model meets the requirement of being greater than 0.75, otherwise it represents that the overall performance of the model is poor.

[0024] In some embodiments, the S3: non-dense vegetation single target detection and identification stage further includes the following steps: S31: Use the raster spatial data conversion library to perform sliding sampling on the orthophoto panchromatic image of the target area to obtain sampled image data, and the sliding sampling detects the confidence level of trees and plants B Tree Greater than or equal to the preset confidence; the sampling sliding step of the sliding sampling is less than the number of pixels of the sample, and the sampling sliding step includes the horizontal sliding step S X and the longitudinal sliding step length S Y , let the size of the sample be Xsample ×Y sample Pixels, 0<horizontal sliding step length S X <X sample , 0<longitudinal sliding step length S Y <Y sample ; S32: Input the sampled image data into the trained deep neural network model for data prediction to obtain multiple recognition detection frames; the center coordinates P of the recognition detection frames m ( x m , y m ) is calculated by the affine transformation matrix T of the orthophoto panchromatic image of the target area, the sliding window of the sliding sampling and the pixel coordinate space relationship of the orthophoto panchromatic image of the target area, and its calculation formula is as follows: In the formula, x m P m The easting value of is in meters and satisfies x m ≥0; y m P m The northing value of is in meters and satisfies y m ≥0; x The pixel horizontal coordinate value of the orthophoto panchromatic image of the target area where the upper left corner pixel of the current sampled image data is located, in pixels. x ≥0; y The pixel ordinate value of the orthophoto panchromatic image of the target area where the upper left corner pixel of the current sampled image data is located, in pixels, and satisfies y ≥0; x 1 is the confidence level B Tree >The tree plant detection result window with preset confidence is located at the leftmost pixel horizontal coordinate within the sampled image data range and meets x 1≥0, unit is pixel; x 2 is the confidence level B Tree >The tree plant detection result window with preset confidence is located at the rightmost pixel horizontal coordinate within the sampled image data range and meets x 2≥0, unit is pixel; y 1 is the confidence level B Tre e> The tree plant detection result window with preset confidence is located at the top pixel vertical coordinate within the sampling image data range and meets y 1≥0, unit is pixel; y 2 is the confidence level B Tree>The tree plant detection result window with preset confidence is located at the bottom pixel vertical coordinate within the sampled image data range and meets y 2≥0, unit is pixel; T 11 is the first element of the first row in the affine transformation matrix T, which means that the upper left corner anchor pixel of the orthophoto panchromatic image of the target area is located at the north distance of the corresponding projection coordinate system and satisfies T 11 ≥0, in meters; T 12 is the second element in the first row of the affine transformation matrix, which means the width of the horizontal pixels of the orthophoto panchromatic image of the target area and satisfies T 12 >0, in meters; T 13 It is the third element in the first row of the affine transformation matrix, which means the horizontal rotation angle of the pixels of the orthophoto panchromatic image of the target area; T 21 is the first element of the last row in the affine transformation matrix, which means that the upper left corner anchor pixel of the orthophoto panchromatic image of the target area is located at the easting of the corresponding projection coordinate system and satisfies T 21 ≥0, in meters; T 22 It is the second element of the last row in the affine transformation matrix, which means the vertical rotation angle of the pixel of the orthophoto panchromatic image of the target area; T 23 is the third element of the last row in the affine transformation matrix, which means the width of the vertical pixels of the orthophoto panchromatic image of the target area and satisfies T 23 <0, unit is meter; Identify the area of ​​the detection frame S Box Calculated by the following formula: ; The center coordinate P of the detection frame is identified m ( x m , y m ),area S Box Stored in CSV files, and x m , y m , S Box They are stored as three independent fields and converted into x m , y m Convert it into two-dimensional coordinate points, and finally convert the vector data to obtain geographic vector data.

[0025] In this embodiment, S31 calls the trained deep neural network model to predict remote sensing data, uses the "torch.load" method to load the trained deep neural network model results, and pre-processes the data based on Python's GDAL library. It is usually necessary to obtain the geographic transformation and geographic coordinate system of the remote sensing data in advance to ensure that the image geographic information is not affected. Note that large-scale remote sensing observation data is usually large, so it is necessary to use data segmentation or sliding window forms to predict the data in batches and then splice them. Otherwise, in most cases, it will be affected by computer hardware limitations and the efficiency of model calculation of excessive data, resulting in the inability to operate the recognition calculation of vegetation monomer targets. The present invention adopts an image sliding sampling mode to improve computing efficiency. First, the implementer can customize the function based on the GDAL library to perform small-range sliding sampling from the original image, and note that the sampling sliding step size is often smaller than the number of pixels in the data acquisition range. For example, the sampling sample size is X sample ×Y sample pixels, then its horizontal and vertical sliding step length is S X , S Y Must satisfy 0 X <X sample and 0 Y <Y sample . The confidence level of the detected tree plant results B Tree Meet B Tree ≥0.75, that is, the probability that the model determines that this target is a tree plant is greater than or equal to 75%. In S32, the sampled image data is used as the input, and the loaded deep neural network model is used to predict the data, and a target window of several prediction results is obtained, which is reflected in the result as the horizontal and vertical coordinate values ​​of the pixels of the four corner points of the window in the image. Based on GDAL, the affine transformation matrix T of the original input orthophoto full-color image and the spatial relationship between the pixel coordinates of the sliding window and the original image are obtained, so that the center coordinate P of a certain recognition detection frame can be derived. m ( x m , y m ), in the calculation formula, T 13 , T 22 The value is usually 0, and the unit is decimal degrees. m ( x m , y m ) Get the coordinates of the four corner points of each predicted recognition detection frame, and calculate the area of ​​the recognition detection frame S Box ​​, used to evaluate the size of trees. The predicted result coordinates and the corresponding identification detection box area are stored in a CSV file, and the point X coordinate, point Y coordinate, and point corresponding identification detection box area are stored as three independent fields Point_X, Point_Y, and S_Box in the file, separated by English commas, and converted from the data table to two-dimensional coordinate points through the data table processing function "XYTableToPoint" in the Python-based geographic data processing library ArcPy. The conversion parameters {x_field} and {y_field} correspond to the Point_X and Point_Y fields in the aforementioned data table, respectively. The other parameters except the input and output file paths are kept as default values. Finally, "FeatureClassToShapefile" is used to convert it into a shapefile format geographic vector data with the suffix ".shp".

[0026] In some embodiments, the S4: modeling environment preprocessing stage further includes the following steps: S41: Perform digital terrain modeling using digital elevation model data, wherein the digital elevation model data has the same data range as the orthophoto panchromatic image data; perform vectorization processing on the digital elevation model data using the ArcPy library, output contour lines in the digital elevation model data as polyline vectors, and then perform line smoothing processing on the polyline vectors to obtain input vector line elements; then create an irregular triangulated network TIN, extract three-dimensional surface elements containing elevation information from the irregular triangulated network TIN, and export them as three-dimensional surface element data in a preset format; S42: performing an interpolation operation on the geographic vector data to obtain a three-dimensional point set containing elevation information; S43: creating a constant grid; importing the three-dimensional surface feature data and the constant grid into the three-dimensional modeling software; importing the three-dimensional point set into the three-dimensional modeling software as a data source for calculating the generation position of the single vegetation model; S44: Use the three-dimensional surface feature data to match the height value of the constant raster layer to obtain a basic modeling scene.

[0027] In this embodiment, S41 constructs a three-dimensional digital terrain through a grid or a topographic map, and it is necessary to obtain a digital elevation model (DEM) that is similar to the real ground. First, the present invention models the digital terrain based on the open source DEM data source of ALOS. The data range obtained should be consistent with the orthophoto panchromatic image range initially selected by the present invention. The implementer can cut and adjust the downloaded data, and its coordinate system should be transformed using the coordinate parameters corresponding to the processed area, such as the Gaussian projection coordinate system based on Earth 2000. Secondly, the DEM data is vectorized based on the ArcPy library, and the contour lines of the raster data are obtained by processing using the "Contour" function, and the output is a polyline vector in the ".shp" format, and the {contour_interval} parameter is a required item. The smaller the value, the smaller the elevation spacing between adjacent contour lines. Based on the above output polyline vector, the line is smoothed using ArcPy's "cartography.SmoothLine" function. The {algorithm} parameter recommends the use of "BEZIER_INTERPOLATION", that is, the Bezier curve method between the vertices to obtain smoother contour lines. The {error_option} parameter recommends the use of "RESOLVE_ERRORS" to solve the geometric topological errors of internal intersections or self-intersections of line features. Further, the contour lines after the above smoothing are used as input vector line features, and the irregular triangulated network (TIN) is created using the "ddd.CreateTin" function. The parameters {in_features_1, height_field_1, SF_type_1, ...} received mainly involve the first vector data source (can be multiple) and its height field and constraint type involved in the calculation. They need to be set to the first input contour vector data source, the feature corresponding to the height value field, and the linear constraint type as soft constraint. Finally, the "ddd.TinTriangle" function uses the aforementioned TIN data as the data input source to extract the three-dimensional surface elements containing elevation information, and the data is exported in shapefile format through the data export function "FeatureClassToShapefile". (Note: The expressions in the form of "{……}" involved in the present invention all refer to the input parameter names of the functions in the programming library) S42 uses the three-dimensional terrain to make the two-dimensional plane point data obtain elevation values, so that the vegetation conforms to the actual position distribution in space. The irregular triangulated network and the predicted vegetation coordinate point set shapefile file are used as the first parameter {in_surface} and the second parameter {in_feature_class} input data sources respectively, and the "ddd.InterpolateShape" function in ArcPy is used to interpolate the vegetation coordinate point set to output a three-dimensional point set containing elevation information.In S43, first, the "CreateConstantRaster" function in ArcPy is used to create a constant raster. The parameter {constant_value} is the input constant value, which is generally set to 0; {data_type} corresponds to the output data type, which is generally set to "FLOAT" floating point value; {cell_size} corresponds to the output pixel size, which should consider the balance between output accuracy and data volume, and the value range is generally 0.5-2; {extent} is the output range, which should be the same as the output vector triangulation terrain range; after running the process, a floating point raster with pixel values ​​of 0 and in GeoTIFF format is obtained. Subsequently, the three-dimensional surface feature file in shapefile format and the constant raster in GeoTIFF format are imported into CityEngine software. The constant raster input adopts the Terrain import mode. In the input parameters, the full-color orthophoto of the input end when predicting the deep neural network model is used as the map data source in the import parameters to obtain a better display effect of the terrain model. Finally, the three-dimensional point set data is imported as the data source for calculating the location of the single vegetation model. The vectors (including 3D surface elements and 3D point sets) and raster data layers imported in S44 serve as the data basis for scene model construction, and still require data preprocessing to form a complete modeling basic environment. After selecting the 3D surface element layer, use the "Align terrain to shapes" command to match the height value of the constant raster layer with the 3D surface elements to obtain a digital terrain model (DEM) that conforms to the terrain undulations. In particular, in CityEngine2022.1 and earlier versions, after the aforementioned imported constant raster is selected, the minimum unit resolution can be adjusted as needed. The default value is 1024. The larger the resolution, the finer the resolution, and the richer the details that can be obtained from the matched target 3D surface elements. At the same time, the hardware resources consumed by the calculation will also increase accordingly.

[0028] In some embodiments, the S5: model creation and export phase further includes the following steps: S51: Define the main rule, attribute declaration and parameter interaction panel; use the 3D point set as the main rule scope, use the CGA rule to traverse all selected 3D points, then initialize the 3D point injection rules and perform random instance insertion, instance scaling, and random rotation one by one, and rotate the vegetation around the vertical axis at a random angle. The scaling factor of each single vegetation model instance in any direction f Calculated by the following formula; In the formula, S Mean It is the horizontal cross-sectional area of ​​the outer envelope of the current single vegetation model; T MAXThe upper limit of the allowable floating range; T MIN The lower limit of the allowable floating range; N r is Gaussian white noise, which has a distribution characteristic of mean 0 and variance 1; In the Parameter Interaction panel, declare the panel group, and then adjust the scaling factor f Parameters in the calculation formula are declared as attributes; S52: Selecting a data source for calculating a location for generating a single vegetation model, creating a single vegetation model, and performing visual data modification; S53: Exporting the single vegetation model after the visualization data modification into a three-dimensional model exchange format to obtain a vegetation environment model scene in a target data format.

[0029] In this embodiment, S51 first writes CGA rules, which are mainly divided into three major parts: main rules, attributes, and parameter interaction panels. The main rule specifies the name of the parent node corresponding to the starting rule, which is generally named StartRule or Lot, and its definition ends with "-->"; the main rule subsequently connects any geometric element operation or calculation function and subsequent derived child nodes, and the calculations are performed in sequence. In this solution, based on the imported three-dimensional point elements as the main rule scope, the CGA rule will traverse all selected three-dimensional points, initialize its main rules, and perform subsequent geometric operations one by one. For subsequent geometric operations, this solution adopts three steps of random instance insertion + instance scaling + random rotation. The instance insertion function is "i(file_path)", which is mainly used to insert an external 3D model based on the point initialized by the main rule, where "file_path" is the tree model file path in the resource library, and can also be other third-party model file paths; the instance scaling function is "s(scale_x, scale_z, scale_y)", which is mainly used to scale the size based on the outer envelope of the inserted model. scale_x, scale_z, and scale_y are the scaling factors in the horizontal, vertical, and vertical directions, respectively, and their values ​​are floating point numbers; the instance rotation function is "r(angle_x,angle_z, angle_y)", which is mainly used to perform angular rotation based on the model insertion base point. angle_x, angle_z, and angle_y are the rotation angles of the horizontal axis, horizontal axis, and vertical axis around the base point, respectively. This scheme only performs random angle rotation on the vegetation around the vertical axis, and the random angle can be generated by the "rand" function. It is worth noting that when scaling the vegetation, the "S_Box" field carried by the aforementioned input point set should be considered, that is, the area of ​​the identification detection box S Box, the scaling factor in each direction of the size of each vegetation model instance f After the calculation, the model will be more natural after the scaling transformation and random rotation based on the area dimension. The attribute declaration is a predefined process based on the relevant parameters manipulated by the rule part in the CGA code. The present invention uses the scaling factor f The parameters in the calculation formula are all considered as predefined attributes. The "file_path" input parameter of the "i function" is the file path, and the value type is a string. Therefore, multiple model paths can be predefined as attributes and placed into an array as elements for random calls. T MAX and T MIN When it is predefined as a static attribute, the initial value is directly assigned as needed; S Box and S Mean It is a dynamic attribute with a predefined initial value of 0. When the data needs to be called, the field name is used as the right value and the attribute name is used as the left value through the equal sign to assign the value. The parameter interaction panel in the "Inspector" parameter indicator of the software provides modelers with a space for custom expansion of the non-coding parameter interaction interface. It is necessary to use "@Group(group_name)" to declare the panel group before the attribute declaration in the CGA code, where the "group_name" parameter is the group name, and is combined in the form of "@Order(order_sequence)" + "interaction type". The former declaration and the "order_sequence" parameter specify the location of the attribute interaction field, and the latter specifies the type of attribute interaction. After that, the corresponding attribute declaration and initialization can be followed. After the CGA code in S52 is written, all three-dimensional point sets are selected, and the model is automatically generated using the "Generate models of selected shapes" command of CityEngine to obtain a preliminary single-unit vegetation model. The implementer observes and modifies the visual data of the specific vegetation model based on the interactive panel of the "Inspector" parameter indicator, including its scaling, rotation parameters, vegetation type path, etc. In S53, all results are exported to a common three-dimensional model exchange format, such as ".obj", ".FBX", etc. After selecting the above-mentioned generated and adjusted single vegetation model and DEM layer in the "Scene" layer display window, use the "ExportModels" command to output the selected model and obtain the final scene model result in the target data format, that is, the vegetation environment model scene in the target data format.

[0030] In some embodiments, the deep neural network model is a Faster-RCNN model.

[0031] In some embodiments, the mapping analysis software is ArcGIS Pro software.

[0032] The present invention uses open source orthophoto full-color satellite images in visible light mode and open source 12.5-meter resolution digital elevation data sources as the data basis for non-intensive vegetation identification and positioning. Compared with the orthophoto aerial photography acquisition mode with higher data resolution, the technical path of the present invention has a higher degree of freedom because the data covers almost the entire world. The implementer can choose any area to acquire data without having to collect data through on-site aerial photography. Based on the above key point solution, the standardized labels and training data annotation processing flow in the ArcGIS Pro desktop geographic information system platform are used for deep learning training data annotation and export. Compared with other deep learning label annotation and processing processes and tools, the present invention is more applicable to tasks with geographic location information as the core.

[0033] For the case of non-intensive vegetation coverage, the present invention uses the Faster-RCNN architecture with a relatively balanced construction difficulty and model performance as the modeling basis. It can bring good recognition accuracy even when the resolution of orthophoto full-color image data is slightly inferior to that of field aerial photography data. And based on the ONNX general deep learning model format, it brings a certain cross-platform migration capability to the vegetation recognition model, so that technicians do not have to repeatedly build task models for non-intensive vegetation monomer recognition.

[0034] The present invention selects GDAL and pytorch to jointly call the output result of the recognition model, and successively performs sliding sampling target recognition of the input data map and calculation of the center coordinates of the detection window based on the affine transformation matrix, so that the vegetation recognition frame can be seamlessly output as a geographic coordinate table centered on the target monitoring result frame while carrying geographic information, and the corresponding detection frame area can be brought out as field information to obtain the area value of the detection target, laying the foundation for subsequent model size estimation.

[0035] Determining the spatial coordinates of basic terrain and vegetation modeling through terrain modeling and vegetation coordinate vector interpolation helps to restore the actual vegetation distribution. Contour line smoothing based on Bezier curves and terrain surface reconstruction based on triangulated networks improves the quality of DEM data, thereby improving the accuracy of the spatial distribution of the non-intensive vegetation scene model involved in the present invention.

[0036] The vegetation model is automatically generated based on the CGA rule, and the model size and direction are rotated based on the recognition window area, assisted by random white noise, so that the generated vegetation model is closer to the natural state. At the same time, an interface for later manual adjustment of the intervention model parameters is reserved, so that the present invention can improve a certain degree of freedom while taking into account efficiency.

[0037] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A digital twin vegetation environment modeling method based on remote sensing images and CGA rules, Features: The following steps are involved: S1: Image data preprocessing stage, obtaining satellite image data, including orthophoto panchromatic image data and digital elevation model data; performing data preprocessing on the satellite image data to obtain first satellite image data; performing target detection mask calibration on the first satellite image data to obtain training data and export it; S2: Target detection model training and tuning phase, building a deep neural network model and using training data to train it to obtain a trained deep neural network model; S3: In the stage of detecting and identifying single tree targets in non-dense vegetation, the trained deep neural network model is used to detect and identify single tree targets in the orthophoto panchromatic image of the target area, obtain the detection frame, and convert the vector data into geographic vector data; S4: Modeling environment preprocessing stage, building the modeling environment and performing environmental preprocessing on the modeling environment to obtain the basic modeling scene; S5: Model creation and export stage, based on geographic vector data and basic modeling scenarios, a single vegetation model is created to obtain a vegetation environment model scene.

2. According to claim 1, the digital twin vegetation environment modeling method based on remote sensing images and CGA rules, Features: The S1: image data preprocessing stage further includes the following steps: S11: Select a data source and obtain satellite image data from the data source; S12: importing the satellite image data into the map drawing and analysis software; using the mosaicking tool to perform mosaicking operation on the satellite image data, and using the coordinate conversion method to resample the satellite image data, thereby transforming it into the first satellite image data based on the standard Gauss projection coordinate system; S13: Load the first satellite image data into the map workspace of the map drawing and analysis software and select the target data set; then perform mask mapping on the target data set and perform rectangular frame mapping on the orthophoto panchromatic image data to obtain training data.

3. According to claim 1, the digital twin vegetation environment modeling method based on remote sensing images and CGA rules, Features: The S2: target detection model training and tuning phase also includes the following steps: S21: Use the cross entropy loss function and regression loss function to determine the training loss of the deep neural network model, and use the SGD optimizer combined with momentum and weight decay to optimize the training process of the deep neural network model; S22: Adjust the hyperparameters to achieve a first preset quantitative index; test the deep neural network model, and perform training and optimization according to the test results until the deep neural network model reaches a second preset quantitative index to obtain a trained deep neural network model.

4. According to claim 1, the digital twin vegetation environment modeling method based on remote sensing images and CGA rules, Features: The S3: non-dense vegetation single-unit target detection and recognition stage further includes the following steps: S31: Use the raster spatial data conversion library to slide sample the orthophoto panchromatic image of the target area to obtain sampled image data, and the sliding sampling detects the confidence level of trees and plants B Tree Greater than or equal to the preset confidence; the sampling sliding step of the sliding sampling is less than the number of pixels of the sample, and the sampling sliding step includes the lateral sliding step S X and the longitudinal sliding step length S Y , let the size of the sample be X sample ×Y sample Pixels, 0<horizontal sliding step length S X <X sample , 0<longitudinal sliding step length S Y <Y sample ; S32: Input the sampled image data into the trained deep neural network model for data prediction to obtain multiple recognition detection frames; the center coordinates P of the recognition detection frames m ( x m , y m ) is calculated by the affine transformation matrix T of the orthophoto panchromatic image of the target area, the sliding window of the sliding sampling and the pixel coordinate space relationship of the orthophoto panchromatic image of the target area, and its calculation formula is as follows: In the formula, x m P m The easting value of is in meters and satisfies x m ≥0; y m P m The northing value of is in meters and satisfies y m ≥0; x The pixel horizontal coordinate value of the orthophoto panchromatic image of the target area where the upper left corner pixel of the current sampled image data is located, in pixels. x ≥0; y The pixel ordinate value of the orthophoto panchromatic image of the target area where the upper left corner pixel of the current sampled image data is located, in pixels, and satisfies y ≥0; x 1 is the confidence level B Tree >The tree plant detection result window with preset confidence is located at the leftmost pixel horizontal coordinate within the sampled image data range and meets x 1 ≥0, unit is pixel; x 2 is the confidence level B Tree >The tree plant detection result window with preset confidence is located at the rightmost pixel horizontal coordinate within the sampled image data range and meets x 2 ≥0, unit is pixel; y 1 is the confidence level B Tre e> The tree plant detection result window with preset confidence is located at the topmost pixel vertical coordinate within the sampled image data range and meets y 1≥0, unit is pixel; y 2 is the confidence level B Tree >The tree plant detection result window with preset confidence is located at the bottom pixel vertical coordinate within the sampled image data range and meets y 2 ≥0, unit is pixel; T 11 is the first element of the first row in the affine transformation matrix T, which means that the upper left corner anchor pixel of the orthophoto panchromatic image of the target area is located at the north distance of the corresponding projection coordinate system and satisfies T 11 ≥0, in meters; T 12 is the second element in the first row of the affine transformation matrix, which means the width of the horizontal pixels of the orthophoto panchromatic image of the target area and satisfies T 12 >0, in meters; T 13 It is the third element in the first row of the affine transformation matrix, which means the horizontal rotation angle of the pixels of the orthophoto panchromatic image of the target area; T 21 is the first element of the last row in the affine transformation matrix, which means that the upper left corner anchor pixel of the orthophoto panchromatic image of the target area is located at the easting of the corresponding projection coordinate system and satisfies T 21 ≥0, in meters; T 22 It is the second element of the last row in the affine transformation matrix, which means the vertical rotation angle of the pixel of the orthophoto panchromatic image of the target area; T 23 is the third element of the last row in the affine transformation matrix, which means the width of the vertical pixels of the orthophoto panchromatic image of the target area and satisfies T 23 <0, unit is meter; Identify the area of ​​the detection frame S Box Calculated by the following formula: ； The center coordinate P of the detection frame is identified m ( x m , y m ),area S Box Stored in CSV files, and x m , y m , S Box They are stored as three independent fields and converted into x m , y m Convert it into two-dimensional coordinate points, and finally convert the vector data to obtain geographic vector data.

5. According to claim 4, the digital twin vegetation environment modeling method based on remote sensing images and CGA rules, Features: The S4: modeling environment preprocessing stage further includes the following steps: S41: Perform digital terrain modeling using digital elevation model data, wherein the digital elevation model data has the same data range as the orthophoto panchromatic image data; perform vectorization processing on the digital elevation model data using the ArcPy library, output contour lines in the digital elevation model data as polyline vectors, and then perform line smoothing processing on the polyline vectors to obtain input vector line elements; then create an irregular triangulated network TIN, extract three-dimensional surface elements containing elevation information from the irregular triangulated network TIN, and export them as three-dimensional surface element data in a preset format; S42: performing an interpolation operation on the geographic vector data to obtain a three-dimensional point set containing elevation information; S43: creating a constant grid; importing the three-dimensional surface feature data and the constant grid into the three-dimensional modeling software; importing the three-dimensional point set into the three-dimensional modeling software as a data source for calculating the generation position of the single vegetation model; S44: Use the three-dimensional surface feature data to match the height value of the constant raster layer to obtain a basic modeling scene.

6. According to claim 5, the digital twin vegetation environment modeling method based on remote sensing images and CGA rules, Features: The S5: model creation and export phase also includes the following steps: S51: Define the main rule, attribute declaration and parameter interaction panel; use the 3D point set as the main rule scope, use the CGA rule to traverse all selected 3D points, then initialize the 3D point injection rules and perform random instance insertion, instance scaling, and random rotation one by one, and rotate the vegetation around the vertical axis at a random angle. The scaling factor of each single vegetation model instance in any direction f Calculated by the following formula; In the formula, S Mean It is the horizontal cross-sectional area of ​​the outer envelope of the current single vegetation model; T MAX The upper limit of the allowable floating range; T MIN The lower limit of the allowable floating range; N r is Gaussian white noise, which has a distribution characteristic of mean 0 and variance 1; In the Parameter Interaction panel, declare the panel group, and then adjust the scaling factor f Parameters in the calculation formula are declared as attributes; S52: Selecting a data source for calculating a location for generating a single vegetation model, creating a single vegetation model, and performing visual data modification; S53: Exporting the single vegetation model after the visualization data modification into a three-dimensional model exchange format to obtain a vegetation environment model scene in a target data format.

7. The digital twin vegetation environment modeling method based on remote sensing images and CGA rules according to any one of claims 1 to 6, Features: The deep neural network model is the Faster-RCNN model.

8. The digital twin vegetation environment modeling method based on remote sensing images and CGA rules according to claim 2 or 4, Features: The map drawing and analysis software is ArcGIS Pro software.

Images