Emergency water supply big data multi-source spatial data rapid searching method

Abstract

The invention provides an emergency water supply big data multi-source spatial data rapid searching method. The method comprises the following steps: collecting emergency water supply related data and sorting and summarizing the data; carrying out data hierarchical design and spatial data storage; carrying out Spark spatial data query calculation; generating a spatial data list; and generating spatial preview data. According to the emergency water supply big data multi-source spatial data rapid query method provided by the invention, two key technical means of supporting multiple storage types by spatial data query preview and distributed query calculation analysis based on big data are adopted, so that the problems existing in processing of a large amount of spatial data in the prior art are solved; and a more efficient and flexible method is provided for storage, query, calculation and analysis of spatial data.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-source spatial data processing, and particularly to a method for quickly viewing multi-source spatial data of emergency water supply big data. Background Art

[0002] With the advent of the big data era, the data sources relied on by emergency water supply services have become increasingly rich and diverse. From Internet of Things monitoring devices, remote sensing satellite images to various business management systems, a vast amount of data is being generated all the time. These data not only have a huge scale but also are complex and variable in type, covering structured data (such as water plant data, water diversion project data, etc.), semi-structured data (such as log files, XML documents, etc.) and unstructured data (such as satellite images, video materials, etc.). Under such a data flood, as a key element for emergency water supply business decision-making, the storage and efficient query of spatial data have become major challenges that need to be solved urgently.

[0003] The existing technical solution is to store spatial data in a spatial database and rely on the spatial functions of the spatial database for data query processing. This solution can meet the storage and query requirements of spatial data to a certain extent, especially when the data volume is relatively small and the data type is relatively single. However, when faced with the massive data and multi-type storage requirements in the big data era, this solution seems inadequate.

[0004] Although a spatial database can store spatial data, its scalability and flexibility are limited. When the data volume grows rapidly, the spatial database may not be able to effectively support the horizontal expansion of data, resulting in a storage bottleneck. At the same time, when processing massive spatial data, the computing performance of the spatial database often becomes a bottleneck. Especially when performing complex spatial queries and analyses, the computing efficiency and response speed will both drop significantly. Also, the spatial database has obvious deficiencies in supporting the storage and query of multi-type data and cannot effectively process semi-structured data and unstructured data. In addition, migrating massive data into the spatial database and maintaining its normal operation requires a large amount of time and technical costs.

[0005] The existing spatial data storage and query solution based on a spatial database can no longer meet the growing storage and query requirements in the big data era. Therefore, a new technical means is urgently needed to solve the problems of multi-type storage, efficient query and preview of big data volume spatial data.

[0006] The Chinese invention patent application with the publication number of CN117453731A discloses a multi-source data query system and a multi-source data query method. The system includes: a coordinator, which is used to parse the query elements of the query statement sent by the user side received, form query sub-statements in units of data sources for the parsed query elements, and allocate each query sub-statement to the corresponding sub-working node in this coordinator. The sub-working node forms each query sub-statement into a task to be sent down respectively and sends down the task to be sent down; a connector, which is used to match with the data source, receive the task to be sent down for the data source matched by this connector, and send the task to be sent down to the matched data source; the coordinator is also used to receive the data obtained by the matched data source in response to the query of the task to be sent down and return the data to the user side. For multi-source data query, a connector is matched for each data source. The disadvantage of this method is that there is no automatic recovery mechanism when the connector node fails. In large-scale emergency events, the system may crash due to overload.

[0007] The Chinese invention patent application with the publication number of CN107958085A discloses an agricultural resource multi-source spatial data sharing platform and its query method, including the following steps: 1) The human-computer interaction interface receives the query request input by the user and sends it to the query request module; 2) The query request module discriminates the input query request information and sends the query request that meets the discrimination conditions to the query request analysis module; 3) The query request analysis module processes the received query request, generates a single-layer spatial data source query request with additional spatial query constraint conditions and sends it to the query execution module; 4) The query execution module queries the database according to the received single-layer spatial data source query request and sends the query result to the query information integration module; 5) The query information integration module is used to integrate the query results and send them to the human-computer interaction interface in the form of a list or a map for display. The disadvantage of this method is that streaming data needs to be stored in the database before it can be queried, resulting in a delay of minutes. Summary of the Invention

[0008] In order to solve the above technical problems, a rapid multi-source spatial data browsing method for emergency water supply big data proposed by the present invention solves the problems existing in the prior art when dealing with large amounts of spatial data by adopting two key technical means of spatial data query preview supporting multiple storage types and big data distributed query calculation analysis, and provides a more efficient and flexible method for the storage, query and calculation analysis of spatial data.

[0009] The object of the present invention is to provide a rapid multi-source spatial data browsing method for emergency water supply big data, including collecting relevant information on emergency water supply and sorting it out, and further including the following steps: Step 1: Data hierarchical design and spatial data storage; Step 2: Perform Spark spatial data query and calculation; Step 3: Generate a spatial data list; Step 4: Generate spatial preview data.

[0010] Preferably, the emergency water supply related materials include key information of important reservoirs, and the key information includes at least one of water supply quantity prediction, water demand prediction of water plants, urban water supply planning map, water plant distribution map, emergency water supply plan and water transfer plan.

[0011] Preferably in any of the above solutions, the sorting and summarization includes the following sub-steps: Step 01: Through meticulous classification, screening and duplicate removal of the emergency water supply related materials, eliminate irrelevant or redundant information; Step 02: Extract the core and valuable content; Step 03: Structurally process the sorted materials and make tables and charts.

[0012] Preferably in any of the above solutions, Step 1 includes classifying and layering the data according to the source and attribute differences of the spatial data.

[0013] Preferably in any of the above solutions, the data classification includes dividing the data into the following categories: 1) The first type of data: Massive spatial data; 2) The second type of data: Spatial data with a small amount of data or that needs to be frequently queried and updated; 3) The third type of data: Specific non-spatial data or data that needs to be quickly accessed; 4) The fourth type of data: Specific file format data or data that needs to be stored for a long time.

[0014] Preferably in any of the above solutions, store the first type of data in a distributed storage system.

[0015] Preferably in any of the above solutions, store the second type of data in a spatial database.

[0016] Preferably in any of the above solutions, store the third type of data in a non-spatial database.

[0017] Preferably in any of the above solutions, store the fourth type of data in file form or use object storage.

[0018] Preferably in any of the above solutions, the spatial field is set as a string type when stored, rather than a specific spatial field type.

[0019] Preferably in any of the above solutions, Step 2 includes the following sub-steps: Step 21: Access and load spatial data from any data source according to the data reading ability of Spark; Step 22: After the data loading is completed, Spark processes the spatial data in a distributed manner; Step 23: Use Spark to perform spatial calculation operations on the spatial data; Step 24: Return the calculation result to the user.

[0020] Preferably, in any of the above solutions, step 21 includes uniformly loading the data distributed in different storage systems onto each Spark node.

[0021] Preferably, in any of the above solutions, the spatial calculation operations include at least one of thinning, distance calculation, range query, intersection judgment, and inclusion relationship analysis.

[0022] Preferably, in any of the above solutions, step 23 includes that when the user submits a query request through Spark SQL, Spark will parse the query statement and filter and calculate the data according to the query conditions.

[0023] Preferably, in any of the above solutions, the calculation result is a set of spatial data after filtering and calculation, or statistical information or visualization charts generated according to the user's needs.

[0024] Preferably, in any of the above solutions, step 3 includes the following sub-steps: Step 31: Obtain a Dataset from the calculation result; Step 32: Parse the Dataset and extract the spatial information therein; Step 33: Convert the Dataset to a List <string>in the form of; Step 34: Traverse each record in the Dataset, and convert each record into a string representation according to the type and format requirements of the spatial data; Step 35: Add the converted string to the List <string>A complete list of spatial data is formed.

[0025] Preferably, in any of the above solutions, the step 4 includes the following sub-steps: Step 41: According to List <string>Generate corresponding spatial layers based on the data therein; Step 42: Preview and display the generated spatial layers on the application side.

[0026] In any of the above solutions, preferably, the spatial layer includes a point layer, a line layer, and a surface layer.

[0027] In any of the above solutions, preferably, step 41 includes, for the point layer, drawing corresponding point markers on the map according to the coordinates.

[0028] In any of the above solutions, preferably, step 41 further includes, for the line layer, drawing line segments connecting the points.

[0029] In any of the above solutions, preferably, step 41 further includes, for the surface layer, drawing a polygon area according to the boundary coordinates.

[0030] In any of the above solutions, preferably, step 41 further includes, during the process of generating the layer, setting the style of the layer as needed.

[0031] The present invention proposes a method for quickly viewing multi-source spatial data of emergency water supply big data, avoiding the migration process of spatial data, thereby saving additional storage hardware, time, and labor costs; through the distributed query engine and spatial data index optimization, the query efficiency is significantly improved when processing large amounts of spatial data.

[0032] Spark represents a fast and general distributed computing engine designed specifically for large-scale data processing.

[0033] Spark SQL represents a module in Spark for processing structured data.

[0034] Dataset represents a distributed dataset in Spark. Brief Description of the Drawings

[0035] Figure 1 It is a flowchart of a preferred embodiment of the method for quickly viewing multi-source spatial data of emergency water supply big data according to the present invention.

[0036] Figure 2 It is a schematic diagram of an embodiment of the string type storage of the method for quickly viewing multi-source spatial data of emergency water supply big data according to the present invention.

[0037] Figure 3 It is a schematic diagram of an embodiment of the surface data preview of the method for quickly viewing multi-source spatial data of emergency water supply big data according to the present invention. Detailed Embodiments

[0038] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0039] Embodiment 1 As Figure 1 shown, a method for quickly viewing multi-source spatial data of emergency water supply big data performs step 100, collecting and organizing relevant information on emergency water supply. The relevant information on emergency water supply includes key information of important reservoirs, and the key information includes at least one of water supply volume prediction, water demand prediction of water plants, urban water supply planning maps, water plant distribution maps, emergency water supply plans, and water transfer plans.

[0040] The organization and induction include the following sub-steps: Perform step 101, through meticulous classification, screening, and duplicate removal of the relevant information on emergency water supply, eliminating irrelevant or redundant information; Perform step 102, extracting core and valuable content; Perform step 103, performing structured processing on the organized information, and making tables and charts.

[0041] Perform step 110, data hierarchical design and spatial data storage, including classifying and hierarchically processing data according to the source and attribute differences of spatial data. Data classification includes dividing data into the following types: 1) The first type of data: massive spatial data; 2) The second type of data: spatial data with a small amount of data or requiring frequent querying and updating; 3) The third type of data: specific non-spatial data or data requiring quick access; 4) The fourth type of data: data in a specific file format or data requiring long-term storage.

[0042] Store the first type of data in a distributed storage system.

[0043] Store the second type of data in a spatial database.

[0044] Store the third type of data in a non-spatial database.

[0045] Store the fourth type of data in file form or use object storage.

[0046] The spatial field is set to a string type when stored, rather than a specific spatial field type.

[0047] Perform step 120, performing Spark spatial data query calculation, including the following sub-steps: Execute step 121, access and load spatial data from any data source according to the data reading ability of Spark, and uniformly load the data in different storage systems onto each Spark node.

[0048] Execute step 122. After the data loading is completed, Spark processes the spatial data in a distributed manner.

[0049] Execute step 123, perform spatial calculation operations on the spatial data using Spark. The spatial calculation operations include at least one of thinning, distance calculation, range query, intersection judgment, and inclusion relationship analysis. When the user submits a query request through SparkSQL, Spark will parse the query statement and filter and calculate the data according to the query conditions.

[0050] Execute step 124, return the calculation result to the user. The calculation result is a set of spatial data after filtering and calculation, or statistical information or visualization charts generated according to the user's needs.

[0051] Execute step 130 to generate a list of spatial data, including the following sub-steps: Execute step 131 to obtain a Dataset from the calculation result; Execute step 132 to parse the Dataset and extract the spatial information therein; Execute step 133 to convert the Dataset to a List <string>in the form of; Execute step 134, traverse each record in the Dataset, and convert each record into a string representation according to the type and format requirements of the spatial data; Execute step 135, add the converted string to the List <string>In it, a complete spatial data list is formed.

[0052] Execute step 140 to generate spatial preview data, including the following sub-steps: Execute step 141 according to List <string>The data in the ,generates the corresponding spatial layer, which includes a point layer, a line layer and a surface layer.

[0053] For point layers, corresponding point markers are drawn on the map according to the coordinates.

[0054] For line layers, draw line segments connecting the points.

[0055] For area layers, polygonal areas are drawn based on the bounding coordinates.

[0056] When generating layers, style the layers as needed.

[0057] Execute step 142 to preview and display the generated spatial layer on the application side.

[0058] Example 2 The present invention provides a method for previewing spatial data based on big data multi-storage type query, aiming to solve the problems of the existing technology in processing large amounts of spatial data, such as single storage type, low query efficiency, and inability to support large-scale data calculation and analysis.

[0059] The present invention breaks the limitation that traditional spatial data is only stored in spatial databases, and expands the storage types to multiple storage methods such as non-spatial databases, text type storage, and object storage. By designing a unified data access interface and conversion mechanism, seamless query and preview of spatial data in different storage types are achieved. When a user initiates a spatial data query request, the system first parses the request to determine the data type and storage location of the required query. Then, through the corresponding data access interface, the data is retrieved from the corresponding storage type. Finally, the retrieved data is formatted and preprocessed to meet subsequent preview and display requirements. The present invention adopts a layered architecture design, including a data access layer, a data processing layer, and a data display layer. The data access layer is responsible for interacting with different storage types, the data processing layer is responsible for data conversion and preprocessing, and the data display layer is responsible for visually displaying the processed data.

[0060] As the underlying layer of the architecture, the data access layer mainly focuses on interacting with diverse data storage types to achieve data access and extraction. These storage types may include, but are not limited to, relational databases (such as MySQL, PostgreSQL), NoSQL databases (such as MongoDB), distributed file systems (such as Hadoop HDFS), and spatial databases (such as PostGIS), etc. The data processing layer is the core of the architecture, responsible for preprocessing tasks such as cleaning, transforming, aggregating, and spatial analysis on the received raw data to meet the requirements of subsequent display and analysis. The data display layer is the user-system interaction interface, responsible for visually displaying the processed data, enabling users to intuitively understand the data and make decisions quickly.

[0061] In addition, the present invention also adopts big data distributed query computing technology. By constructing a distributed computing cluster, it realizes efficient spatial computing and analysis of data volumes in the millions, tens of millions, hundreds of millions, or even larger scales. Utilizing the parallel processing capabilities of distributed computing frameworks (such as Hadoop, Spark, etc.), it can quickly process large-scale data and improve the efficiency of querying and computing. When large-scale spatial data computing and analysis are required, the system first splits the computing tasks into multiple subtasks and distributes these subtasks to each node in the distributed computing cluster. Then, each node executes the subtasks in parallel and returns the results to the system. Finally, the system summarizes and processes the returned results to obtain the final computing and analysis results.

[0062] Based on the distributed computing cluster, the present invention constructs a processing framework specifically for spatial data computing and analysis. This framework includes a task splitting module, a task scheduling module, a computing execution module, and a result summarizing module, etc., which can achieve efficient processing and analysis of large-scale data. The task splitting module supports splitting complex spatial data computing and analysis tasks into multiple subtasks for parallel processing in the distributed computing cluster; the task scheduling module evenly distributes tasks to each node through intelligent algorithms, avoiding node overload and node idleness, and improving resource utilization efficiency; the computing execution module utilizes a distributed computing framework (such as Hadoop MapReduce) to achieve parallel processing and accelerate the computing process; the result summarizing module summarizes and integrates the results obtained from executing subtasks on each computing node to form the final analysis report or visual display.

[0063] Improvements or inventive points of the present invention: There are two main improvements in the present invention. One is to expand the storage type of spatial data from a single spatial database to multiple storage methods, thus greatly improving the storage flexibility and query efficiency of data. This inventive point enables the present invention to better adapt to the diversity of data types and the hugeness of data volume in the big data era. The other is to apply the big data distributed query computing technology to the calculation and analysis of spatial data, thus realizing the efficient processing of large-scale data. This inventive point enables the present invention to process massive data that cannot be dealt with by traditional methods, providing strong support for the in-depth mining and intelligent analysis of spatial data.

[0064] The function or role of each technical means in the technical solution: The present invention adopts a unified data access interface and conversion mechanism to achieve seamless access and conversion of spatial data in different storage types, improving the query efficiency and preview effect of data. It supports the access of multiple storage types and the format conversion of data, ensuring the consistency and accuracy of data during query and preview.

[0065] The present invention adopts the big data distributed query computing technology and utilizes the parallel processing ability of the distributed computing framework to achieve the efficient query and calculation analysis of large-scale data. It supports the splitting, distribution, parallel processing and result summarization of data, improving the efficiency and accuracy of query and calculation. At the same time, through constructing a special processing framework, the in-depth mining and intelligent analysis of spatial data are realized.

[0066] By adopting two key technical means, namely, multi-storage type supported spatial data query preview and big data distributed query calculation analysis, the present invention solves the problems existing in the prior art when dealing with spatial data with a large amount of data, providing a more efficient and flexible method for the storage, query and calculation analysis of spatial data.

[0067] Compared with traditional methods, the present invention avoids the migration process of spatial data, thus saving additional storage hardware, time and labor costs. Through the distributed query engine and spatial data index optimization, the query efficiency of the present invention is significantly improved when dealing with spatial data with a large amount of data. Experimental data show that when dealing with 100 million spatial data records, the query response time of the present invention is reduced by about 70% compared with the traditional spatial database, shortening from several hours to dozens of minutes. The design of multi-storage type adaptation enables the present invention to easily meet the spatial data query requirements under different storage environments, improving the flexibility and scalability of the system. The data preview interface provides an intuitive and convenient way for users to view query results, greatly enhancing the user experience.

[0068] When traditional spatial databases handle large amounts of spatial data, they often face the problem of low query efficiency due to limited computing resources. At the same time, the data migration process also increases the enterprise's operating costs and time costs. In contrast, by introducing big data technology and a distributed computing architecture, the present invention not only solves the bottleneck problem of large-volume data queries but also eliminates the data migration step, thereby reducing the overall cost of the enterprise while ensuring query efficiency. In addition, the multi-storage type adaptation layer and data preview interface of the present invention also greatly improve the flexibility and user experience of the system.

[0069] Embodiment 3 Based on big data means, the present invention directly reads data from different storage types and loads it into memory for spatial queries without the need for data migration work. When facing massive data calculations, based on the capabilities of big data distributed computing, the data in memory is distributed to different machine nodes for queries and then aggregated, which can meet the increasing data query calculations.

[0070] The process steps of the present invention are as follows: (1) Collect relevant information on emergency water supply and organize and summarize it. In the scenario of emergency water supply services, in order to build the ability to preview spatial data queries with multiple storage types based on big data, various data materials closely related to emergency water supply are widely collected. These materials cover key information such as the available water volume prediction of important reservoirs, the water demand prediction of water plants, urban water supply planning maps, water plant distribution maps, emergency water supply plans, and water transfer plans. Through meticulous classification, screening, and duplicate removal, irrelevant or redundant information is eliminated, and core and valuable content such as emergency response measures and early warning release and response rules is extracted. Subsequently, using professional data management tools, we structured these organized materials, creating tables and charts to visually display data relationships and trends, providing support and a basis for data layer design and spatial data storage.

[0071] (2) Data layer design and spatial data storage In the scenario of emergency water supply services, based on the spatial data from various sources collected in step (1), data layer design and spatial data storage are completed.

[0072] First, according to the source and attribute differences of spatial data, the data is classified and stratified. This process ensures the reasonable organization and efficient management of the data. Specifically, for massive spatial data at the GB / TB / PB level, it is selected to be stored in a distributed storage system, such as HDFS (Hadoop Distributed File System). The distributed storage system has powerful data storage and expansion capabilities, can cope with the storage requirements of massive data, and at the same time ensure the reliability and availability of the data. This storage method is suitable for scenarios with huge data volume and relatively low query frequency, such as long-term monitoring data, historical data, etc. Secondly, for spatial data with relatively small data volume or that needs to be frequently queried and updated, according to the characteristics of the data, it is selectively stored in a spatial database (such as PostGis). The spatial database is specifically used to store and manage spatial data and provides rich spatial query and analysis functions. This storage method is suitable for scenarios that require frequent spatial queries, analysis, and data updates, such as real-time water supply network monitoring data, emergency water supply plan data, etc. Thirdly, for certain specific non-spatial data or data that needs to be accessed quickly, it is selected to be stored in a non-spatial database, such as Redis, ClickHouse, etc. These databases have the characteristics of high performance and low latency and can meet application scenarios with high requirements for data real-time performance. Finally, for certain specific file format data or data that needs to be stored for a long time, it is selected to be stored in file form or object storage (such as Minio, Oss, etc.). File storage and object storage provide flexible data storage and management methods and can meet the needs of different application scenarios.

[0073] (III) Spark Spatial Data Query and Calculation In the scenario of facing the emergency water supply business, when users need to perform filtering queries on the massive spatial data in storage, such as querying the water supply line planning map of a certain city or finding key information such as water diversion and regulation projects, sluices, water plants, and reservoirs, the present invention relies on the powerful capabilities of Spark to implement an efficient spatial data query and calculation method.

[0074] First, rely on Spark's data reading ability to access and load spatial data from any data source. These data may be scattered in different storage systems, such as HDFS, spatial databases, non-spatial databases, etc., and they are uniformly loaded onto each Spark node. Secondly, after the data is loaded, Spark will process the spatial data in a distributed manner. It should be noted that in order to simplify the data processing process and improve efficiency, as Figure 2 shown, the spatial fields are set as string types when stored, rather than specific spatial field types. This design makes the data more flexible and efficient during the loading and calculation processes.

[0075] Again, Spark utilizes its built-in rich computing capabilities to perform spatial computing operations on data, such as data thinning, distance calculation, range query, intersection judgment, and inclusion relationship analysis. These calculations are based on Spark's distributed computing power, so it can handle massive amounts of spatial data and the computing speed is extremely fast. Specifically, when a user submits a query request through Spark SQL, Spark will parse the query statement and filter and calculate the data according to the query conditions. For example, if a user wants to query the water plant data within a certain city, Spark SQL will parse out the query conditions and perform the filtering operation distributively on each Spark node to find the data that meets the conditions.

[0076] Finally, Spark returns the calculated results to the user. These results may be a set of spatial data after filtering and calculation, or statistical information or visualization charts generated according to the user's needs, etc.

[0077] (IV) Generating Spatial Preview Data After completing the Spark spatial data query calculation and presenting the data results in the form of a Dataset, these data cannot be directly used by the application side to generate preview spatial data. In order to display clear and intuitive spatial layers, such as point, line, and surface layers, on the application side, we need to perform a series of conversion processes on the Dataset.

[0078] Obtain the Dataset object from the results after Spark's calculation. This Dataset contains the spatial data after spatial calculation, filtering, and other operations. Parse the Dataset and extract the spatial information in it, such as the coordinates of points, the node sequence of lines, and the boundaries of surfaces. Since the data in the Dataset may be stored in a complex data structure, we need to convert it into a data format that the application side can directly process. Here, we choose to convert the Dataset to a List <string>in the form. Traverse each record in the Dataset and convert each record into a string representation according to the type (point, line, surface) and format requirements of the spatial data. This string may contain the coordinates, attribute information, etc. of the spatial data. Add the converted string to the List <string>In it, a complete spatial data list is formed.

[0079] (V) Generate spatial preview data According to List <string>The data in [the relevant context] is used to generate corresponding spatial layers. This involves parsing strings into spatial geometric objects and creating layers based on these objects. For point layers, it may be necessary to draw corresponding point markers on the map according to the coordinates; for line layers, it is necessary to draw line segments connecting the points; as Figure 3 shown, for polygon layers, it is necessary to draw polygon areas according to the boundary coordinates. During the process of generating layers, the layers can also be styled as needed, such as color, line thickness, fill style, etc., to meet different visualization requirements.

[0080] The generated spatial layers are previewed and displayed on the application side. Users can view these layers through the application interface and interact with them, such as zooming, panning, click query, etc. To improve the user experience and query efficiency, data caching, indexing and other mechanisms of the layers can also be implemented on the application side to quickly respond to users' query requests.

[0081] To better understand the present invention, the above has been described in detail in combination with specific embodiments of the present invention, but it is not a limitation of the present invention. Any simple modification made to the above embodiments based on the technical essence of the present invention still belongs to the scope of the technical solution of the present invention. Each embodiment in this specification focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.< / string> < / string> < / string> < / string> < / string> < / string> < / string> < / string> < / string>

Claims

1. A method for quickly viewing multi-source spatial data of emergency water supply big data, including collecting relevant information on emergency water supply and organizing and summarizing it, characterized in that, The following steps are also included: Step 1: Data layer design and spatial data storage; Step 2: Perform Spark spatial data query and calculation; Step 3: Generate a spatial data list, including the following sub-steps: Step 31: Get Dataset from the calculation results; Step 32: Parse the Dataset and extract the spatial information therein; Step 33: Convert the Dataset to a List <string> form;< / string> Step 34: Traverse each record in the Dataset and convert each record into a string representation according to the type and format requirements of the spatial data; Step 35: Add the converted string to the List <string> In the , a complete list of spatial data is formed;< / string> Step 4: Generate spatial preview data.

2. The rapid browsing method for multi-source spatial data of emergency water supply big data according to claim 1, wherein, The emergency water supply related information includes key information of important reservoirs, and the key information includes at least one of water supply forecast, water plant water demand forecast, urban water supply planning map, water plant distribution map, emergency water supply plan and water diversion plan.

3. The rapid multi-source spatial data browsing method for emergency water supply big data according to claim 2, characterized in that The arranging and summarizing comprises the following sub-steps: Step 01: Classify, screen and remove duplicates of the emergency water supply related information to eliminate irrelevant or redundant information; Step 02: Extract the core and valuable content; Step 03: Structuralize the organized data and create tables and charts.

4. The rapid multi-source spatial data browsing method for emergency water supply big data according to claim 3, characterized in that The step 1 includes classifying and stratifying the data according to the sources and attribute differences of the spatial data.

5. The rapid multi-source spatial data browsing method for emergency water supply big data according to claim 4, wherein Data classification involves classifying data into the following categories: 1) The first type of data: massive spatial data; 2) Second type of data: spatial data with small data volume or that needs to be frequently queried and updated; 3) The third type of data: specific non-spatial data or data that requires fast access; 4) The fourth type of data: data in a specific file format or data that needs to be preserved for a long time.

6. The rapid browsing method for multi-source spatial data of emergency water supply big data according to claim 5, characterized in that, The first type of data is stored in a distributed storage system.

7. The rapid browsing method for multi-source spatial data of emergency water supply big data according to claim 6, characterized in that, The third type of data is stored in a non-spatial database.

8. The rapid multi-source spatial data browsing method for emergency water supply big data according to claim 7, characterized in that The fourth type of data is stored in a file format or in an object format.

9. The rapid browsing method for emergency water supply big data multi-source spatial data according to claim 8, characterized in that, Spatial fields are stored as string types rather than as specific spatial field types.

10. The rapid multi-source spatial data browsing method for emergency water supply big data according to claim 9, characterized in that The step 2 includes the following sub-steps: Step 21: Access and load spatial data from any data source using Spark's data reading capabilities. Step 22: After the data is loaded, Spark processes the spatial data in a distributed manner; Step 23: Using Spark to perform spatial computing operations on the spatial data; Step 24: Return the calculated result to the user.

Images