Urban stormwaterlogging simulation data processing method based on distributed computing

Abstract

The present invention discloses a method for processing urban rainstorm waterlogging simulation data based on distributed computing, which includes the following steps: uploading a digital surface model to a control node server, the control node server splitting the digital surface model into data chunks with the same number as the computing node servers according to the number of computing node servers, and distributing N data chunks to N computing node servers for processing; the computing node servers completing simulation calculations according to the rainstorm waterlogging model tool and feeding back the results to the node server; the control node server summarizing and completing the fusion and splicing of the rainstorm waterlogging simulation results according to the processing results of each computing node server. The present invention overcomes the problem of high time consumption of massive geographical data in the application of urban rainstorm waterlogging simulation and calculation, utilizes the rich computing resources provided by the computer cluster, realizes the parallel preprocessing of geographical data and the collection of calculation results, and solves the most time-consuming calculation link in waterlogging simulation.

Description

Technical Field

[0001] This invention relates to the field of urban hydrology, and in particular to a method for processing urban rainstorm and flood simulation data based on distributed computing. Background Technology

[0002] The theoretical basis for urban flood simulation lies in the urban hydrological cycle and the physical mechanisms of hydrodynamics. Compared to natural watersheds, the dense human activities within cities bring greater complexity and uncertainty to hydrological and hydrodynamic processes.

[0003] Traditional urban flood models, such as the SWMM, STORM, and Wallingford models, are primarily used for one-dimensional hydrodynamic simulations of urban drainage networks. Their main limitation is their inability to simulate two-dimensional surface water accumulation processes. Some models have coupled two-dimensional hydrodynamic modules, leading to next-generation one-dimensional / two-dimensional coupled urban flood models such as PC-SWMM (an upgraded version of SWMM), InfoWorks (an upgraded version of Wallingford), and MIKE-Urban. These models use the output of the one-dimensional module as the input to the two-dimensional module, enabling simulation of the entire urban flood process. However, these models require irregular grids to construct urban terrain, resulting in complex modeling and less than ideal simulation results.

[0004] Emergency management of urban flooding has become a hot research area both domestically and internationally. With increasingly higher accuracy and richer variety of geographic data, the scale of input data for numerical simulations of urban rainstorms and flooding is rapidly increasing, and the timeliness of computation cannot meet actual production requirements. Although the computing performance of single machines has improved, it still cannot keep up with the rapidly growing data scale. Methods supporting distributed computing are necessary to significantly improve computing performance in order to apply numerical simulations of urban rainstorms and flooding to real-world urban disaster reduction and prevention applications. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method for processing urban rainstorm and waterlogging simulation data based on distributed computing, so as to overcome the problem of high time consumption in applying massive geographic data to urban rainstorm and waterlogging simulation calculations. By utilizing the rich computing resources provided by computer clusters, parallel preprocessing of geographic data and aggregation of calculation results can be achieved, thus solving the most time-consuming calculation link in waterlogging simulation.

[0006] To achieve the above objectives, the present invention employs the following:

[0007] The urban rainstorm and flooding simulation data processing method based on distributed computing provided by this invention includes the following steps:

[0008] S1: Upload the digital land surface model to the control node server. The control node server divides the digital land surface model into data blocks equal to the number of computing node servers based on the number of computing node servers, and distributes the N data blocks to the N computing node servers for processing.

[0009] S2: The computing node server completes the simulation calculation based on the rainstorm and urban flooding model tool and feeds back the results to the node server;

[0010] S3: The control node server, based on the processing results of each computing node server, summarizes and merges the simulation results of the rainstorm and urban flooding.

[0011] In implementation, step S1 involves a digital surface model that includes high-precision urban topographic data, urban drainage zoning data, and urban high-density precipitation prediction data. The digital surface model is segmented using QGIS technology to divide the raster file into blocks. The specific steps for segmentation are as follows:

[0012] S11. Converting river contour features to polygons: Using ArcGIS toolbox, extract the intersection surfaces of features and export the data in shapefile format.

[0013] S12, Section Grouping: Grouping and classifying the cross-sections of water areas that need to be divided;

[0014] S13. Terrain Data Masking and Clipping: Based on Python, call the QGIS masking and clipping method, input the digital land surface model and water surface, and export the raster file in .asc format.

[0015] In implementation, in step S1: after receiving the simulation command, the control node server processes the input digital surface model data, including file format conversion, coordinate registration, loading into memory, and conversion into raster form.

[0016] In implementation, in step S2: after receiving the simulation command, the control node server sends the data address to be processed and the storage address to the rainstorm and waterlogging model tool on each computing node server. After receiving the command, the rainstorm and waterlogging model tool performs the corresponding data simulation calculation. After the calculation is completed, it sends a message to the control node server, and the model calculation result processing program on the control node server processes the calculation results accordingly.

[0017] In implementation, the specific steps for fusing and stitching together the simulation results of rainstorm flooding in step S3 are as follows:

[0018] S31. Data stitching of simulation results for rainstorm and urban flooding:

[0019] After receiving the calculation results from various rainstorm and urban flooding model tools, the control node server calls the QGIS algorithm library based on a Python script, inputs the calculation results in .asc format data files and the latitude and longitude range of each partition, and the model algorithm library automatically stitches the data of each calculation result according to the input insertion. Finally, it calls the data conversion library to export the stitched result as an .asc format raster file.

[0020] S32. Data coloring of the simulation results of rainstorm flooding:

[0021] After receiving the stitching results, the processing software uses the Image object shader, combined with the legend data requirements, to parse, color, and export the ASC format file as a TIFF format file.

[0022] S33, Slice of simulation results of rainstorm flooding:

[0023] Use QGIS to input TIFF format image files, input LOD levels, and output tile files;

[0024] S34. Release of simulation results for rainstorm-induced urban flooding:

[0025] The tile files, located in different folders, are used to create virtual directories for each model folder via IIS, and the simulation results are then published.

[0026] In implementation, in step S34, the tile file of the simulation calculation result can be read using the network path "http: / / ip:port number / virtual directory name / level / X / Y.png".

[0027] Unlike traditional single-model processing, this invention improves model processing efficiency by optimizing the scheduling of model processing software through distributed scheduling; unlike traditional TIFF files, this invention uses ASCII raster data layers to improve the calculation speed of simulation software; unlike traditional TIFF format loading, this invention loads large images on the web through pyramid tile tiling, solving the problem of loading large images on the web in seconds. Attached Figure Description

[0028] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0029] Figure 1 This is a schematic diagram of the hardware environment of the present invention;

[0030] Figure 2 This diagram illustrates the processing steps of a high-precision digital surface model according to the present invention.

[0031] Figure 3 A schematic diagram of the distributed scheduling of the present invention is shown;

[0032] Figure 4 This is a schematic diagram illustrating the tile deployment and loading process of the present invention. Detailed Implementation

[0033] To more clearly illustrate the present invention, the following description, in conjunction with preferred embodiments, further clarifies the invention. Those skilled in the art should understand that the specific descriptions below are illustrative rather than restrictive, and should not be construed as limiting the scope of protection of the present invention.

[0034] Typically, the entire process of urban stormwater flooding simulation calculations can be logically divided into three stages. The first stage involves processing the input data (referred to in this invention as a digital surface model, or simply geographic data), including file format conversion, coordinate registration, loading into memory, and conversion to raster format. The second stage involves performing mathematical formula calculations on the raster values, including hydrological and hydrodynamic formulas, and outputting the results to a temporary file. The third stage involves compiling, visualizing, or otherwise applying the output results in the temporary file. While different researchers may use slightly different mathematical formulas in urban stormwater flooding simulations, specifically in the parameter values ​​and the omission of certain physical quantities, these differences are beyond the scope of this application. This application focuses on addressing the data processing methods in the first and third stages.

[0035] Since this application involves data processing methods based on a computer cluster environment, which places relatively high demands on machine performance, the hardware requirements for the control node server and the computing node server are recommended as follows to ensure the system's operational effectiveness:

[0036] I. One control node server, configuration requirements:

[0037] Operating System: Windows Server 2018 or later

[0038] Processor (CPU): 8 cores, clock speed not less than 2.0GHz

[0039] Memory (RAM): at least 32GB

[0040] The control node server has a built-in hard drive of 200GB.

[0041] II. There are N compute node servers. The configuration requirements for each node are as follows:

[0042] Operating System: Ubuntu 16.04 64-bit

[0043] Processor (CPU): 16 cores, clock speed not less than 2.0GHz

[0044] Memory (RAM): at least 64GB

[0045] The compute node server comes with a 200GB hard drive.

[0046] III. The control node and the computing node share storage of no less than 3TB.

[0047] IV. The local area network speed of all nodes must be no less than 1000Mbps.

[0048] The technical solution of this application is as follows:

[0049] S1: Upload the digital land surface model to the control node server. The control node server divides the digital land surface model into data blocks equal to the number of computing node servers based on the number of computing node servers, and distributes the N data blocks to the N computing node servers for processing.

[0050] S2: The computing node server completes the simulation calculation based on the rainstorm and urban flooding model tool and feeds back the results to the node server;

[0051] S3: The control node server, based on the processing results of each computing node server, summarizes and merges the simulation results of the rainstorm and urban flooding.

[0052] In simple terms, the process begins by using GIS vector data (shp) such as rivers, water systems, and drainage networks to segment a high-precision digital surface model (DSM) (ASC). This is then combined with real-time rainfall data to generate a distributed rainfall raster layer (ASC). Next, a distributed rainstorm flooding simulation is performed using a rainstorm flooding modeling tool. Finally, a distributed fusion tool is used to fuse and display the rainstorm flooding simulation results.

[0053] Firstly, the high-precision digital surface model (DSM) (ASC) is segmented using QGIS technology to divide the raster file into blocks. By programming and calling the QGIS library, machine-driven raster data processing can be achieved.

[0054] The specific steps for segmentation are:

[0055] S11. Converting river contour features to polygons: Using ArcGIS toolbox, extract the intersection surfaces of features and export the data in shapefile format.

[0056] S12, Section Grouping: Grouping and classifying the cross-sections of water areas that need to be divided;

[0057] S13. Terrain Data Masking and Clipping: Based on Python, call the QGIS masking and clipping method, input the digital land surface model and water surface, and export the raster file in .asc format.

[0058] Secondly, distributed scheduling utilizes MQTT for model verification tool scheduling. MQTT is a publish / subscribe messaging protocol based on the ISO standard (ISO / IEC PRF 20922). It operates on the TCP / IP protocol suite and is designed for remote devices with low hardware performance and poor network conditions. The control center (control node server) performs real-time dynamic scheduling of model tools. When the control center (control node server) receives a model simulation verification command, it sends the data addresses to be processed and the storage addresses to each model tool. Upon receiving the command, the model tool performs the relevant data simulation calculations. After the calculation is complete, it sends a message to the control center (control node server). The control center (control node server), upon receiving the message, sends a message to the model calculation result processing program for further processing of the calculation results.

[0059] Thirdly, the fusion and splicing of the simulation results of rainstorm and urban flooding involves processing the various distributed simulation results of rainstorm and urban flooding. QGIS is used to merge and export the model files in ASC format of each block according to the latitude and longitude range. Then, programming tools are used in conjunction with custom legends to colorize the simulation results. The colored results are exported in TIF format. Next, the image pyramid tiling algorithm is used to process the tile data that can be loaded on the web. Finally, IIS is used to publish the relevant data as a service.

[0060] The specific steps for merging and stitching together simulation results of rainstorm and urban flooding are as follows:

[0061] S31. Data stitching of simulation results for rainstorm and urban flooding:

[0062] After receiving the calculation results from various rainstorm and urban flooding model tools, the control node server calls the QGIS algorithm library based on a Python script, inputs the calculation results in .asc format data files and the latitude and longitude range of each partition, and the model algorithm library automatically stitches the data of each calculation result according to the input insertion. Finally, it calls the data conversion library to export the stitched result as an .asc format raster file.

[0063] S32. Data coloring of the simulation results of rainstorm flooding:

[0064] After receiving the stitching results, the processing software uses the Image object shader, combined with the legend data requirements, to parse, color, and export the ASC format file as a TIFF format file.

[0065] S33, Slice of simulation results of rainstorm flooding:

[0066] Use QGIS to input TIFF format image files, input LOD levels, and output tile files;

[0067] S34. Release of simulation results for rainstorm-induced urban flooding:

[0068] By creating virtual directories for each model folder using IIS to store tile files in different folders, and publishing the simulation results, the tile files of the simulation results can be read using the network path "http: / / ip:port number / virtual directory name / level / X / Y.png".

[0069] The following section provides a detailed description of the scheme in this application, using specific examples.

[0070] Example

[0071] See Figure 1 As shown, the hardware environment involved mainly includes a B / S system architecture consisting of 9 servers and several clients. One server serves as a distributed scheduling center node, i.e., a control node server, while the other eight servers serve as computing node servers for simulating urban flooding. The clients are used to display the results of the urban flooding simulation.

[0072] First, the TIFF format of the city's high-precision terrain is converted into an .asc format model, and drainage zone .shp files are generated according to the city's water system and drainage network.

[0073] Next, using the QGIS algorithm library, the terrain file is cut into several terrain ASC files according to the drainage zone, and the hourly rainfall data of each weather station is read, based on the rainfall value and the latitude and longitude of the station.

[0074] Next, the QGIS spatial interpolation algorithm is used to generate an .asc format file of the urban distributed rainfall data raster layer.

[0075] Next, the terrain zoning data and distributed rainfall data are submitted to the distributed scheduling center, which is the control node server. After receiving the data, the distributed scheduling center sends it to the rainstorm and waterlogging simulation tools on each computing node server. After receiving the message from the distributed scheduling center, each rainstorm and waterlogging simulation tool performs rainstorm and waterlogging simulation and sends the simulation results back to the distributed scheduling center.

[0076] Then, after receiving the command, the flood control tool on the distributed dispatch center uses QGIS to input the flood result ASC file and the partition SHP file, stitches the flood simulation results into a complete simulation result TIFF map, and uses QGIS's pyramid tile tiling tool to realize the tiling of the rainstorm flood simulation.

[0077] Next, a virtual directory is created using IIS to publish the tile map. All the 3D models and shapefile layers mentioned above are stored on the server side.

[0078] Finally, the flood simulation data published by IIS is loaded in the client to enable fast display and browsing of urban flood simulation results on the web.

[0079] Combination Figure 2 As shown, the steps for slicing urban high-precision terrain based on river networks are as follows:

[0080] A. Data processing of river network centerline

[0081] Load the river network shapefile vector data using ArcGIS. Check if the river network surface is continuous. If not, edit the surface to ensure it is continuous and uninterrupted. Then, use ArcToolbox > Data Management Tools > Features > Polygon to Line to check if the line layer is an open and continuous graphic. Define the projected coordinate system. The method for defining the coordinate system is as follows: ArcToolbox > Data Management Tools > Projection and Transformation > Define Projection. Then, use ArcToolbox > Cartographic Tools > Cartographic Generalization > Extract Centerline to generate the shapefile.

[0082] B. Drainage zone generation

[0083] Load the river network centerline data generated in the previous step, call ArcToolbox—Data Management Tools—Features—Feature to Polygon tool, set the relevant output path and XY tolerance, and generate a shapefile of the distributed drainage zonal polygon.

[0084] C. High-precision urban terrain zoning and tiling

[0085] Using QGIS, load the high-precision urban terrain ASC data and the drainage zone surface SHP file mentioned above. Then, call the toolbox—GDAL—Raster Extraction—Crop Raster by Mask Layer, input the terrain, drainage zone surface, and specified coordinates, and finally export it as ASC format raster terrain data.

[0086] D. Data format conversion

[0087] Open QGIS, load the raster terrain data file generated in the previous step, call the toolbox—GDAL—Raster Projection—Reprojection Tool, input the terrain data file, specify the source CRS, target CRS, select the resampling method nearest neighbor sampling, select the export data format asc and the export directory, and generate distributed terrain asc raster data that can be processed.

[0088] Combination Figure 3As shown, the steps for using a distributed scheduling center (control node server) are as follows:

[0089] a. Import data

[0090] Upload distributed terrain zoning data, drainage zoning data, and distributed rainfall data to the dispatch center software.

[0091] b. Issue simulation calculation commands

[0092] After receiving the simulation command, the dispatch center sends the command to each flood simulation tool via MQTT. The command contains the parameter information that each simulation tool needs to input. After receiving the command, the flood simulation tool automatically performs calculations and processing.

[0093] c. Processing of Rainstorm and Urban Flooding Simulation Results

[0094] After the rainstorm and flooding simulation tool completes the simulation calculation, it automatically sends a simulation success message to the dispatch center. The message includes the simulation result status and the final path of the simulation result. After receiving the command, the dispatch center issues it to the flooding result processing tool. After receiving the command, the flooding result processing tool calls QGIS's toolbox—GDAL—Raster Miscellaneous—Merge, inputs all flooding processing result .asc files, selects the output data type as Float32, sets the pixel value of invalid data to 0, starts processing, and exports the processed data as an .asc file. Then, using coloring software, based on the given legend, it generates a colored TIFF image.

[0095] Combination Figure 4 As shown, the steps for publishing and displaying the slices of the rainstorm and urban flooding simulation results are as follows:

[0096] 1. Generate tile images

[0097] Open QGIS, locate the directory containing the TIFF image of the waterlogging simulation results in your browser, double-click the file to load it in QGIS, and after loading, call QGIS Toolbox—GDAL—Raster Miscellaneous—gdal2tiles. In the interface, select the waterlogging simulation results as the input layer, Mercator projection as the tile segmentation plane, scale level 1-18, average resampling method, WGS84 coordinate system, select the output directory, and click the Process button to start generating the tile file.

[0098] 2. IIS publishes tile data

[0099] Right-click on Computer -> Manage -> Services and Applications -> Internet Information Services (IIS) Manager -> Right-click on Websites -> Add Website. In the prompt box, enter the website name and select the physical path. Right-click to create a new virtual directory. Name it as needed. Select the folder where the flood simulation slice is located as the physical path and set it to allow directory browsing. For model files, IIS cannot recognize them for network publishing, so you need to set the MIME type. Add an item with the .PNG extension and MIME type application / octet-stream. When calling the model using JS, cross-domain issues may occur. In IIS, you need to set "HTTP Response Headers" to allow cross-domain access. Add the following three key-value pairs: Access-Control-Allow-Headers: Content-Type, api_key, Authorization; Access-Control-Allow-Origin: *; Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS.

[0100] The tile file of the simulation results can be read using the network path "http: / / ip:port number / virtual directory name / level / X / Y.png".

[0101] In summary, according to the technology and method described in the embodiments of this application, by performing distributed computing, the original single processing software is distributed and scheduled, and the city is divided into several regions according to partitions. The amount of data processed by each simulation tool is greatly reduced. The processing time of more than one hour can be optimized to 5 minutes to process a city. The processing results are processed through pyramid tiles, and the web terminal is rendered in seconds.

[0102] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. For those skilled in the art, other variations or modifications can be made based on the above description. It is impossible to exhaustively list all the implementation methods here. All obvious variations or modifications derived from the technical solutions of the present invention are still within the protection scope of the present invention.

Claims

1. A data processing method for urban stormwater flooding simulation based on distributed computing, characterized in that, Includes the following steps: S1: Upload the digital land surface model to the control node server. The control node server divides the digital land surface model into data blocks equal to the number of computing node servers based on the number of computing node servers, and distributes the N data blocks to the N computing node servers for processing. S2: The computing node server completes the simulation calculation based on the rainstorm and urban flooding model tool and feeds back the results to the node server; S3: The control node server summarizes and stitches together the simulation results of rainstorm and urban flooding based on the processing results of each computing node server; In step S1, the digital surface model includes high-precision urban topographic data, urban drainage zoning data, and urban high-density precipitation prediction data. The segmentation of the digital surface model is achieved by dividing the raster file into blocks using QGIS technology. The specific steps of segmentation are as follows: S11. Converting river contour features to polygons: Using ArcGIS toolbox, extract the intersection surfaces of features and export the data in shapefile format. S12, Section Grouping: Grouping and classifying the cross-sections of water areas that need to be divided; S13. Terrain Data Masking and Cropping: Based on Python, call the QGIS masking and cropping method, input digital land surface model and water surface, and export .asc format raster file; Specifically, in step S3, the steps for fusing and stitching together the simulation results of rainstorm flooding are as follows: S31. Data stitching of simulation results for rainstorm and urban flooding: After receiving the calculation results from various rainstorm and urban flooding model tools, the control node server calls the QGIS algorithm library based on a Python script, inputs the calculation results in .asc format data files and the latitude and longitude range of each partition, and the model algorithm library automatically stitches the data of each calculation result according to the input insertion. Finally, it calls the data conversion library to export the stitched result as an .asc format raster file. S32. Data coloring of the simulation results of rainstorm flooding: After receiving the stitching results, the processing software uses the Image object shader, combined with the legend data requirements, to parse, color, and export the ASC format file as a TIFF format file. S33, Slice of simulation results of rainstorm flooding: Use QGIS to input TIFF format image files, input LOD levels, and output tile files; S34. Release of simulation results for rainstorm-induced urban flooding: The tile files located in different folders are used to create virtual directories for each model folder via IIS, and the simulation results are published. In step S34, the tile file of the simulation result can be read using the network path "http: / / ip:port number / virtual directory name / level / X / Y .png".

2. The urban rainstorm and flooding simulation data processing method based on distributed computing according to claim 1, characterized in that, In step S1: After receiving the simulation command, the control node server processes the input digital land model data, including file format conversion, coordinate registration, loading into memory, and conversion into raster format.

3. The urban rainstorm and flooding simulation data processing method based on distributed computing according to claim 1, characterized in that, In step S2: After receiving the simulation command, the control node server sends the data address to be processed and the storage address to the rainstorm and waterlogging model tool on each computing node server. After receiving the command, the rainstorm and waterlogging model tool performs the corresponding data simulation calculation. After the calculation is completed, it sends a message to the control node server, and the model calculation result processing program on the control node server processes the calculation result accordingly.

Images