A spatial data processing method, device and medium
Through the concurrent analysis technology of nifi data flow templates and processor clusters, the performance bottlenecks and security issues of spatial big data processing in the GIS field are solved, fast and efficient data transmission and encrypted storage are achieved, and data processing efficiency and system reliability are improved.
Patent Information
- Application Number
- CN202411195269.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-29
AI Technical Summary
In the case of large amount of data, spatial data is passed into HDFS through a single thread, making it difficult to complete quickly and efficiently, resulting in a performance bottleneck in the GIS field of spatial big data processing.
The task is automatically started using the nifi data flow template, the task is split and concurrently analyzed through the nifi processor cluster, the data processing is accelerated using a multi-core processor, and the processed data is encrypted and transmitted to the distributed file system in packets.
It significantly improves data processing speed and security, solves the performance bottlenecks and security issues of spatial big data processing in the GIS field, and improves data processing efficiency and system reliability.
Smart Images

Figure CN118708641B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a spatial data processing method, device and medium. Background Art
[0002] With the development of GIS (Geographic Information System) technology, GIS has been widely used in the fields of land, water conservancy, transportation, agriculture, and smart cities. As the amount of GIS data increases, traditional GIS tools encounter performance bottlenecks when processing spatial data.
[0003] With the development and maturity of big data technology, ArcGIS GA came into being. As a GIS platform integrated with big data processing capabilities, ArcGIS GA effectively solves the performance problem of spatial big data processing by adopting HDFS (Hadoop Distributed File System) as distributed storage and Spark for distributed computing.
[0004] However, in practical applications, the processed spatial data is transferred to HDFS in a single-threaded manner, so it is difficult to complete the transfer of spatial data quickly and efficiently when the data volume is large. Summary of the invention
[0005] The embodiments of the present application provide a spatial data processing method, device and medium for solving the following technical problem: spatial data is transferred to HDFS in a single-threaded manner, so that it is difficult to complete the transfer of spatial data quickly and efficiently when the data volume is large.
[0006] The present application embodiment adopts the following technical solutions:
[0007] The embodiment of the present application provides a spatial data processing method. The method includes: responding to a task scheduling instruction, starting a Nifi task based on a preset Nifi data flow template, and passing the task parameters to a Nifi processor cluster; establishing a connection between the task parameters and a spatial database through the Nifi processor cluster; constructing a task element set based on the layer structure corresponding to the Nifi task, splitting the Nifi task based on the task element set and the Nifi processor cluster to form multiple subtask data streams; performing concurrent analysis on multiple subtask data streams through the Nifi processor cluster, and concurrently processing the data in the spatial database based on the instructions corresponding to the multiple subtasks; dividing the processed data, encrypting the divided data in groups, and passing the encrypted data to a distributed file system; detecting the task data corresponding to the acquired Nifi task, and determining that the spatial data is successfully imported if the detection passes.
[0008] The embodiment of the present application realizes the automatic start of tasks through the preset nifi data flow template, without manual intervention, and improves work efficiency. The nifi processor cluster can perform concurrent analysis on multiple subtask data streams, fully utilize the computing power of multi-core processors, and significantly speed up data processing. The processed data is encrypted in groups to ensure the security of data transmission and storage, prevent the leakage of sensitive information, and use distributed file systems such as HDFS to store encrypted data, provide high availability and fault tolerance, and further ensure data security. The embodiment of the present application solves the performance bottlenecks and security issues of spatial big data processing in the GIS field through automation, concurrent processing, data encryption and other methods, and improves data processing efficiency and system reliability.
[0009] In one implementation of the present application, in response to a task scheduling instruction, a Nifi task is started based on a preset Nifi data flow template, specifically including: building a processing task based on task information and task requirements, as well as determined processing layers and detection rules; wherein the task information includes at least one of a task code and a task name, and the task requirement includes at least one of task detection, projection transformation, area calculation, and field filtering; in response to the task scheduling instruction, a processing task corresponding to the task scheduling instruction is determined in a task list; a preset Nifi data flow template is called to create a Nifi task corresponding to the processing task in Nifi; task parameters are determined based on the task name, and the task parameters are sent to the Nifi task to start the Nifi task.
[0010] In one implementation of the present application, a connection between task parameters and a spatial database is established through a NiFi processor cluster, specifically including: receiving task parameters through a TaskConfigProcessor processor, and converting the task parameters into a data stream for transmission; receiving a data stream through a GetFeatureProcessor processor, and determining a corresponding spatial database based on the data stream, and establishing a connection with the spatial database; determining parameter keywords based on the task parameters, and querying task data in a preset cache library based on the parameter keywords; if no corresponding data is found, querying task data in a spatial database through a preset interface and parameter keywords; wherein the task data is the number of layer elements and the maximum ObjectID corresponding to the task parameters; storing the task data in a preset cache library, and writing the task data into the data stream.
[0011] In one implementation of the present application, based on the layer structure corresponding to the Nifi task, a task element set is constructed, and based on the task element set and the Nifi processor cluster, the Nifi task is split to form multiple sub-task data streams, specifically including: based on the constructed connection, determining the layer structure corresponding to the Nifi task; determining multiple first elements corresponding to the first layer in the layer structure, and determining multiple second elements respectively associated with the multiple first elements in the next layer; based on the types of the multiple first elements, constructing multiple reference element sets, and dividing the elements corresponding to the layer structure into multiple reference element sets based on the correlation; based on the number of elements corresponding to the multiple reference element sets, dynamically dividing the multiple reference element sets to obtain multiple task element sets; based on the multiple task element sets, splitting the Nifi task into multiple sub-tasks through the Nifi processor cluster, and forming multiple sub-task data streams.
[0012] In one implementation of the present application, multiple reference element sets are dynamically divided based on the number of elements corresponding to each of the multiple reference element sets to obtain multiple task element sets, specifically including: taking the preset number of elements corresponding to each subtask as a reference value; based on the reference value, dynamically dividing the multiple reference element sets to obtain multiple element subsets, and assigning a corresponding index identifier to each element subset; wherein the index identifier includes a start index and an end index; mapping the reference element set, the index identifier and the element subset, and storing the mapping relationship; generating multiple task element sets based on the multiple element subsets, and marking the mapping relationship based on the generated task element sets.
[0013] In one implementation of the present application, a nifi processor cluster is used to perform concurrent analysis on multiple subtask data streams, so that based on the instructions corresponding to the multiple subtasks, the data in the spatial database are concurrently processed, specifically including: the KafkaConsumer processor in the nifi processor cluster is used to listen to the messages in the distributed message queue system, and when a new message is listened to, the new message is passed to the SdeToShpProcessor processor; the new message is parsed by the SdeToShpProcessor processor to obtain the instructions corresponding to the new message; based on the instructions corresponding to the new message, the corresponding spatial query attributes and spatial query conditions are determined; wherein the spatial query conditions include spatial range and spatial relationship; a spatial query statement is constructed based on the spatial query attributes and the spatial query conditions, so as to query in the spatial database through the spatial query statement and obtain the corresponding spatial data; the acquired spatial data is processed accordingly, and the processed data is encapsulated into the required format.
[0014] In one implementation of the present application, the processed data is divided, the divided data is encrypted in groups, and the encrypted data is transmitted to a distributed file system, specifically including: dividing the processed data into first reference data and second reference data; performing a first encryption process on a preset first subkey and the first reference data to obtain third reference data, and performing a second encryption process on a preset second subkey and the second reference data to obtain fourth reference data; partially replacing the third reference data based on the preset first spare data, and partially replacing the fourth reference data based on the preset second spare data; performing a cross-talk between the replaced third reference data and the replaced fourth reference data; or process to obtain fifth reference data; based on the preset third spare data, partially replace the fifth reference data, and encrypt the replaced fifth reference data with the preset third subkey to obtain sixth reference data; based on the preset fourth spare data, partially replace the sixth reference data, XOR the replaced fourth reference data with the replaced sixth reference data to obtain seventh reference data, replace the seventh reference data, and encrypt the replaced seventh reference data with the preset fourth subkey; until the number of subkeys involved in the encryption process is greater than the preset number of subkeys to complete the encryption of the working data; pass the encrypted data to the distributed file system.
[0015] In one implementation of the present application, the task monitoring data corresponding to the acquired Nifi task is detected, specifically including: performing topology detection on the data flow transmission process corresponding to the Nifi task; wherein the topology detection includes at least one of hanging point detection, overlap detection and intersection detection; performing encoding detection on the field corresponding to the Nifi task; wherein the encoding detection includes at least one of the value range detection and dictionary value detection of the field; performing metadata detection on the field corresponding to the Nifi task; wherein the metadata detection includes at least one of the field name detection, field type detection and field missing detection.
[0016] An embodiment of the present application provides a spatial data processing device, including: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: respond to a task scheduling instruction, start a Nifi task based on a preset Nifi data flow template, and pass the task parameters to a Nifi processor cluster; establish a connection between the task parameters and a spatial database through the Nifi processor cluster; construct a task element set based on a layer structure corresponding to the Nifi task, and split the Nifi task based on the task element set and the Nifi processor cluster to form a plurality of subtask data streams; perform concurrent analysis on the plurality of subtask data streams through the Nifi processor cluster, so as to concurrently process the data in the spatial database based on the instructions corresponding to the plurality of subtasks respectively; divide the processed data, encrypt the divided data in groups, and pass the encrypted data to a distributed file system; detect the task data corresponding to the acquired Nifi task, and determine that the spatial data import is successful if the detection passes.
[0017] A non-volatile computer storage medium provided in an embodiment of the present application stores computer executable instructions, and the computer executable instructions are configured to: respond to a task scheduling instruction, start a Nifi task based on a preset Nifi data flow template, and pass the task parameters to a Nifi processor cluster; establish a connection between the task parameters and a spatial database through the Nifi processor cluster; construct a task element set based on a layer structure corresponding to the Nifi task, and split the Nifi task based on the task element set and the Nifi processor cluster to form a plurality of subtask data streams; perform concurrent analysis on the plurality of subtask data streams through the Nifi processor cluster, so as to concurrently process the data in the spatial database based on the instructions corresponding to the plurality of subtasks; divide the processed data, encrypt the divided data in groups, and pass the encrypted data to a distributed file system; detect the task data corresponding to the acquired Nifi task, and determine that the spatial data import is successful if the detection passes.
[0018] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: The embodiments of the present application realize the automatic start of tasks through the preset nifi data flow template, without manual intervention, and improve work efficiency. The nifi processor cluster can perform concurrent analysis on multiple subtask data streams, make full use of the computing power of multi-core processors, and significantly speed up data processing. The processed data is encrypted in groups to ensure the security of data transmission and storage, prevent the leakage of sensitive information, and use distributed file systems such as HDFS to store encrypted data, provide high availability and fault tolerance, and further ensure data security. The embodiments of the present application solve the performance bottlenecks and security issues of spatial big data processing in the GIS field through automation, concurrent processing, data encryption and other methods, and improve data processing efficiency and system reliability. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:
[0020] Figure 1 A flow chart of a spatial data processing method provided in an embodiment of the present application;
[0021] Figure 2 A schematic diagram of the structure of a spatial data processing device provided in an embodiment of the present application.
[0022] Reference numerals:
[0023] 200: spatial data processing device, 201: processor, 202: memory. DETAILED DESCRIPTION
[0024] Embodiments of the present application provide a spatial data processing method, device, and medium.
[0025] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.
[0026] The technical solution proposed in the embodiment of the present invention is described in detail below with reference to the accompanying drawings.
[0027] Figure 1 A flow chart of a spatial data processing method provided in an embodiment of the present application is as follows: Figure 1 As shown, the spatial data processing method includes the following steps:
[0028] Step 101, respond to the task scheduling instruction, start the NiFi task based on the preset NiFi data flow template, and pass the task parameters to the NiFi processor cluster.
[0029] In one embodiment of the present application, a processing task is constructed based on task information and task requirements, as well as determined processing layers and detection rules; wherein the task information includes at least one of a task code and a task name, and the task requirements include at least one of task detection, projection transformation, area calculation, and field filtering. In response to a task scheduling instruction, a processing task corresponding to the task scheduling instruction is determined in a task list. A preset nifi data flow template is called to create a nifi task corresponding to the processing task in nifi. Task parameters are determined based on the task name, and the task parameters are sent to the nifi task to start the nifi task.
[0030] Specifically, Apache Nifi is an easy-to-use, powerful and reliable data processing and distribution system. Apache Nifi is designed for data flow. It supports highly configurable data routing, transformation and system mediation logic of the instruction graph. The key features of Nifi include: web-based user interface, high scalability, process templates, security and data tracking.
[0031] Specifically, respond to the task scheduling instruction to create a processing task. To create a processing task, you need to fill in and select the corresponding task information on the task management page. Fill in the task code and task name, select whether the task will perform quality assessment, projection transformation, area calculation, field filtering, etc., and add the layers to be processed and the corresponding assessment rules.
[0032] Further, select the processing task to be scheduled in the task list. When starting the task, the system will call the pre-defined Nifi data flow template, create a Nifi task in Nifi, and then query the task parameters according to the task name, pass them to the Nifi task and start the task. After the task is started, the task parameters are passed to the Nifi processor cluster.
[0033] Step 102: Establish a connection between the task parameters and the spatial database through the NiFi processor cluster.
[0034] In one embodiment of the present application, task parameters are received through the TaskConfigProcessor processor, and the task parameters are converted into a data stream for transmission. The data stream is received through the GetFeatureProcessor processor, and the corresponding spatial database is determined based on the data stream, and a connection with the spatial database is established. Parameter keywords are determined based on the task parameters, and task data is queried in the preset cache library based on the parameter keywords. If the corresponding data is not found, the task data is queried in the spatial database through the preset interface and parameter keywords; wherein the task data is the number of layer elements and the maximum ObjectID corresponding to the task parameters. The task data is stored in the preset cache library, and the task data is written to the data stream.
[0035] Specifically, the task parameters are received through the TaskConfigProcessor processor, which includes relevant information about the geospatial data to be processed, such as layer name, query conditions, etc. The received task parameters are converted into data streams, such as JSON, XML, etc., so as to be transmitted and processed in the system. The data stream is received through the GetFeatureProcessor processor, and the data stream is parsed to identify the spatial database that needs to be operated. Based on the information in the data stream, a connection to the corresponding spatial database, such as PostGIS, GeoServer, etc., is established.
[0036] Further, keywords for query are extracted from the task parameters, including layer name, geographic area, time range, etc. These keywords are used to query whether the required task data already exists in the preset cache library. Among them, the cache library in the embodiment of the present application is usually used to store frequently accessed data to reduce database query pressure.
[0037] Furthermore, if the required data is not found in the cache, a query is established with the spatial database through a preset interface, such as a REST API, a database query statement, etc., using parameter keywords as query conditions. The target data of the query usually includes the number of layer features, that is, the number of geographic features contained in the layer, and also includes the maximum ObjectID, which is a unique identifier used to identify the last geographic feature in the layer. The queried task data is stored in the preset cache for quick access in the future. The queried task data is written to the original data stream so that subsequent processors can use this data to continue processing.
[0038] Step 103: Based on the layer structure corresponding to the NiFi task, a task element set is constructed. Based on the task element set and the NiFi processor cluster, the NiFi task is split to form multiple subtask data flows.
[0039] In one embodiment of the present application, based on the constructed connection, the layer structure corresponding to the NiFi task is determined. Multiple first elements corresponding to the first layer are determined in the layer structure, and multiple second elements respectively associated with the multiple first elements are determined in the next layer. Based on the types of the multiple first elements, multiple reference element sets are constructed, and the elements corresponding to the layer structure are divided into multiple reference element sets based on the relevance. Based on the number of elements corresponding to the multiple reference element sets, the multiple reference element sets are dynamically divided to obtain multiple task element sets. Based on the multiple task element sets, the NiFi task is split into multiple subtasks through the NiFi processor cluster, and multiple subtask data streams are formed.
[0040] Specifically, Nifi first obtains the geospatial data source through the processor cluster, and uses the processor cluster to parse the layer structure information in the data source. The layer structure is usually defined in one or more files, which describe how the data is organized in layers and what types of geographic features each layer contains. Through Nifi's stream file processing logic, the first layer is identified from the parsed layer structure.
[0041] Further, in the first layer, multiple first elements are determined according to element type or geographical location, etc. In the next layer, each first element is traversed to find the second element associated with it. The association in the embodiment of the present application can be based on spatial relationships, such as inclusion relationships, adjacent relationships, attribute matching, etc.
[0042] Further, according to the type attributes of the multiple first elements, they are grouped into different reference element sets. Based on the number of elements in each reference element set, it is dynamically divided into multiple task element sets. Using the distributed processing function of Nifi, each task element set is converted into an independent subtask, and each subtask data stream is sent to different Nifi processor cluster nodes for parallel processing.
[0043] For example, Nifi obtains the GeoJSON file of the city planning through the received task request, parses the GeoJSON file, and identifies the "block" as the first layer. The first few blocks are selected from the "block" layer as the first element. For each block, all the buildings located in the block are found in the "building" layer as the second element. According to the function of the block, such as commercial area, residential area, the blocks are grouped into different reference feature sets. If there are too many blocks in a functional area, they can be further split into smaller task feature sets based on conditions such as block size and building density. Each task feature set is converted into a subtask and processed in parallel by Nifi's distributed processor cluster.
[0044] In one embodiment of the present application, the number of preset elements corresponding to each subtask is used as a reference value. Based on the reference value, multiple reference element sets are dynamically divided to obtain multiple element subsets, and a corresponding index identifier is assigned to each element subset; wherein the index identifier includes a start index and an end index. The reference element set, the index identifier, and the element subset are mapped, and the mapping relationship is stored. Multiple task element sets are generated based on the multiple element subsets, and the mapping relationship is marked based on the generated task element sets.
[0045] Specifically, for each reference feature set, a suitable number of elements is pre-set as a reference value. This value can be determined based on factors such as processing performance, resource constraints, and data balance. Each reference feature set is traversed and dynamically divided according to the total number of elements within it. The division method can be an average distribution, or it can be divided based on the distribution of feature attributes, the continuity of spatial position, etc. The result of the division is to split each reference feature set into multiple feature subsets, and the number of elements in each subset is as close to the reference value as possible.
[0046] Furthermore, a unique index identifier is assigned to each feature subset. This identifier usually includes a start index and an end index, which are used to identify the position range of the subset in the original reference feature set. It should be noted that the generation of index identifiers in the embodiments of the present application is unique and continuous. A mapping relationship is established between the reference feature set, the index identifier, and the feature subset. Based on the divided feature subsets, a corresponding task feature set is generated. Each task feature set will be processed as an independent task or subtask in Nifi. While generating the task feature set, the original mapping relationship is updated or marked, including recording which task feature sets are generated by which feature subsets, and the corresponding relationship between them.
[0047] Step 104: Perform concurrent analysis on multiple subtask data streams through the NiFi processor cluster, so as to concurrently process the data in the spatial database based on the instructions corresponding to the multiple subtasks.
[0048] In one embodiment of the present application, the KafkaConsumer processor in the nifi processor cluster monitors the messages in the distributed message queue system, and when a new message is monitored, the new message is passed to the SdeToShpProcessor processor. The new message is parsed by the SdeToShpProcessor processor to obtain the instruction corresponding to the new message. Based on the instruction corresponding to the new message, the corresponding spatial query attribute and spatial query condition are determined; wherein the spatial query condition includes the spatial range and the spatial relationship. A spatial query statement is constructed based on the spatial query attribute and the spatial query condition, so as to query in the spatial database through the spatial query statement and obtain the corresponding spatial data. The acquired spatial data is processed accordingly, and the processed data is encapsulated into the required format.
[0049] Specifically, the KafkaConsumer processor in nifi is configured to listen for new messages in the Kafka message queue. Kafka is a distributed message queue system, commonly used in big data and high-throughput messaging scenarios. The KafkaConsumer processor will continue to check the specified topic in Kafka, and once a new message arrives, it will trigger the subsequent processing flow. In the nifi process of the embodiment of the present application, when the KafkaConsumer processor captures a new message, it passes the message to the SdeToShpProcessor processor through the configured connection.
[0050] Furthermore, after receiving the message, the SdeToShpProcessor processor will parse the content of the message and extract specific instructions from it. Among them, these instructions include the type of spatial query to be executed, the target attribute of the query, etc. Based on the parsed instructions, the SdeToShpProcessor processor will determine the spatial data attributes to be queried, such as the name and height of the building, etc., as well as the spatial query conditions, such as spatial range, spatial relationship, etc.
[0051] Further, using the determined spatial query attributes and conditions, the SdeToShpProcessor processor will construct a corresponding spatial query statement, such as an SQL statement or a specific GIS query statement, and execute the query in the spatial database through this statement to obtain the required spatial data. After the query results, that is, the spatial data are returned, the SdeToShpProcessor processor will further process the data, such as formatting, converting the coordinate system, etc., and then encapsulate the processed data into the required format for subsequent use or transmission.
[0052] In the embodiment of the present application, nifi tasks process data: TaskConfigProcessor receives task parameters and converts them into data streams and passes them to GetFeatureProcessor. GetFeatureProcessor creates an SDE connection and calculates the number of layer features and the maximum ObjectID (self-incrementing), and writes them into the data stream and passes them to the next processor. LayerBlockProcessor splits the task into multiple tasks according to the configuration, forming multiple data streams written into kafkaPublish processors. These processors only run on the master node of the cluster. All nodes in the cluster run kafkaConsumer processors in a single thread and concurrently run SdeToShpProcessor processors to read the data stream and process the data. It determines when to perform projection transformation, area calculation and field filtering based on the content in the data stream. The data stream is passed to ShpToHDFS for processing, and the data is imported into the HDFS resource pool.
[0053] Step 105: divide the processed data into data, encrypt the divided data in groups, and transfer the encrypted data to the distributed file system.
[0054] In one embodiment of the present application, the processed data is divided into first reference data and second reference data. The preset first subkey and the first reference data are first encrypted to obtain third reference data, and the preset second subkey and the second reference data are second encrypted to obtain fourth reference data. Based on the preset first spare data, the third reference data is partially replaced, and based on the preset second spare data, the fourth reference data is partially replaced. The replaced third reference data and the replaced fourth reference data are XOR-processed to obtain fifth reference data. The fifth reference data is partially replaced based on the preset third spare data, and the replaced fifth reference data is encrypted with the preset third subkey to obtain sixth reference data. The sixth reference data is partially replaced based on the preset fourth spare data, the replaced fourth reference data and the replaced sixth reference data are XOR-processed to obtain seventh reference data, and the seventh reference data is replaced, and the replaced seventh reference data is encrypted with the preset fourth subkey. Until the number of subkeys participating in the encryption process is greater than the preset number of subkeys, the encryption of the working data is completed. The encrypted data is passed to the distributed file system.
[0055] Specifically, the processed data is divided into two parts, namely, the first reference data and the second reference data. The first reference data is encrypted using a preset first subkey. The encryption in the embodiment of the present application can be an AES or RSA encryption method, and the encrypted data is called the third reference data. Similarly, the second reference data is encrypted using a preset second subkey to obtain the fourth reference data. After encryption, some parts of the third reference data and the fourth reference data are replaced. The replacement in the embodiment of the present application is performed using preset spare data. The replacement can replace the data in the middle part, or replace the data at the beginning and the end, the purpose is to further disrupt the original pattern of the data and increase the difficulty of cracking.
[0056] Further, the replaced third reference data is subjected to an XOR operation with the replaced fourth reference data. XOR is a binary operation that compares two binary bits at the same position. If they are the same, the result is 0, and if they are different, the result is 1. The XOR result is a new data block, which is the fifth reference data. Some parts of the fifth reference data are replaced based on the preset third spare data. The replaced fifth reference data is encrypted using the third subkey to obtain the sixth reference data.
[0057] Furthermore, the sixth reference data is partially replaced by presetting the fourth spare data, the replaced fourth reference data is XOR-processed with the replaced sixth reference data to obtain the seventh reference data, the seventh reference data is replaced, and the replaced seventh reference data is encrypted with the preset fourth subkey.
[0058] Furthermore, this process can be repeated as many times as needed, using different subkeys and backup data each time. When the preset subkey number threshold is reached, the encryption process is complete. The encrypted data is securely transferred to the distributed file system for storage.
[0059] Step 106: Detect the task data corresponding to the acquired NiFi task. If the detection passes, it is determined that the spatial data is imported successfully.
[0060] In one embodiment of the present application, a topology detection is performed on the data flow transmission process corresponding to the Nifi task; wherein the topology detection includes at least one of the following: suspension point detection, overlap detection, and intersection detection. An encoding detection is performed on the field corresponding to the Nifi task; wherein the encoding detection includes at least one of the following: value range detection of the field and dictionary value detection. A metadata detection is performed on the field corresponding to the Nifi task; wherein the metadata detection includes at least one of the following: field name detection, field type detection, and field missing detection.
[0061] Specifically, the quality assessment of the embodiment of the present application mainly conducts three aspects of assessment. The topology assessment performs corresponding topology checks based on the transmitted topology rules, and displays the topology check results on the page. The coding check will check whether the field value and value range of the field meet the standards according to the coding rules. The field check will check whether the field of the layer is missing and whether the type of the field meets the standards according to the data standard. The final inspection results will be displayed on the page. Secondly, the task monitoring of the embodiment of the present application mainly monitors the execution of the key nodes of connection creation, data conversion processing, data pooling and quality assessment. For example, the monitoring of SDE to shapefile files can monitor which data block is converted (ObjectId from how much to how much), how long the conversion takes, the execution results and the storage directory of the shapefile.
[0062] Figure 2 This is a schematic diagram of the structure of a spatial data processing device provided in an embodiment of the present application. Figure 2As shown, the spatial data processing device 200 includes: at least one processor 201; and a memory 202 that is communicatively connected to the at least one processor 201; wherein the memory 202 stores instructions that can be executed by the at least one processor 201, and the instructions are executed by the at least one processor 201 so that the at least one processor 201 can: respond to the task scheduling instruction, start the Nifi task based on the preset Nifi data flow template, and pass the task parameters to the Nifi processor cluster; establish a connection between the task parameters and the spatial database through the Nifi processor cluster; construct a task element set based on the layer structure corresponding to the Nifi task, and split the Nifi task based on the task element set and the Nifi processor cluster to form multiple subtask data streams; perform concurrent analysis on the multiple subtask data streams through the Nifi processor cluster, so as to concurrently process the data in the spatial database based on the instructions corresponding to the multiple subtasks; divide the processed data, encrypt the divided data in groups, and pass the encrypted data to the distributed file system; detect the task data corresponding to the acquired Nifi task, and determine that the spatial data import is successful if the detection passes.
[0063] The embodiment of the present application also provides a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions are set to: respond to task scheduling instructions, start Nifi tasks based on preset Nifi data flow templates, and pass task parameters to Nifi processor clusters; establish a connection between task parameters and spatial databases through Nifi processor clusters; build a task element set based on the layer structure corresponding to the Nifi task, and split the Nifi task based on the task element set and the Nifi processor cluster to form multiple subtask data streams; perform concurrent analysis on multiple subtask data streams through the Nifi processor cluster, so as to concurrently process the data in the spatial database based on the instructions corresponding to the multiple subtasks; divide the processed data, encrypt the divided data in groups, and pass the encrypted data to a distributed file system; detect the task data corresponding to the acquired Nifi task, and determine that the spatial data import is successful if the detection passes.
[0064] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0065] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various modifications and changes. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application.
Claims
1. A spatial data processing method, characterized in that: The method comprises: In response to the task scheduling instruction, it starts the NiFi task based on the preset NiFi data flow template and passes the task parameters to the NiFi processor cluster; Establishing a connection between the task parameters and the spatial database through the nifi processor cluster; Based on the layer structure corresponding to the NiFi task, a task element set is constructed, and based on the task element set and the NiFi processor cluster, the NiFi task is split to form multiple subtask data flows; Through the NiFi processor cluster, a plurality of subtask data streams are concurrently analyzed, so as to concurrently process the data in the spatial database accordingly based on the instructions respectively corresponding to the plurality of subtasks; The processed data is divided into data, the divided data is encrypted in groups, and the encrypted data is transmitted to the distributed file system; The task data corresponding to the acquired NiFi task is detected, and if the detection passes, it is determined that the spatial data is imported successfully; The layer structure corresponding to the Nifi task is used to construct a task element set, and the Nifi task is split based on the task element set and the Nifi processor cluster to form multiple subtask data flows, specifically including: Based on the constructed connection, determine the layer structure corresponding to the nifi task; Determine a plurality of first elements corresponding to a first layer in the layer structure, and determine a plurality of second elements respectively associated with the plurality of first elements in a next layer; Based on the types of the first elements, a plurality of reference element sets are constructed, and based on the relevance, the elements corresponding to the layer structure are respectively divided into the plurality of reference element sets; Based on the number of elements respectively corresponding to the multiple reference element sets, the multiple reference element sets are dynamically divided to obtain multiple task element sets; Based on the multiple task element sets, the NiFi task is split into multiple subtasks through the NiFi processor cluster, and multiple subtask data flows are formed.
2. A spatial data processing method according to claim 1, characterized in that: The response task scheduling instruction starts the NiFi task based on the preset NiFi data flow template, specifically including: Based on the task information and task requirements, as well as the determined processing layers and detection rules, construct a processing task; wherein the task information includes at least one of a task code and a task name, and the task requirements include at least one of task detection, projection transformation, area calculation, and field filtering; In response to a task scheduling instruction, determining the processing task corresponding to the task scheduling instruction in a task list; Calling the preset nifi data flow template to create a nifi task corresponding to the processing task in nifi; Task parameters are determined based on the task name, and the task parameters are sent to the NiFi task to start the NiFi task.
3. A spatial data processing method according to claim 1, characterized in that: The connection between the task parameters and the spatial database is established through the nifi processor cluster, specifically including: The task parameters are received through the TaskConfigProcessor processor, and the task parameters are converted into a data stream for transmission; The data stream is received by a GetFeatureProcessor processor, and a corresponding spatial database is determined based on the data stream, and a connection is established with the spatial database; Determine a parameter keyword based on the task parameter, and perform task data query in a preset cache library based on the parameter keyword; In the case where no corresponding data is found, the task data is queried in the spatial database through the preset interface and the parameter keyword; wherein the task data is the number of layer elements and the maximum ObjectID corresponding to the task parameter; The task data is stored in the preset cache library, and the task data is written into the data stream.
4. A spatial data processing method according to claim 1, characterized in that: The dynamically dividing the multiple reference element sets based on the number of elements respectively corresponding to the multiple reference element sets to obtain multiple task element sets specifically includes: The number of preset elements corresponding to each subtask is used as a reference value; Based on the reference values, the multiple reference element sets are dynamically divided to obtain multiple element subsets, and a corresponding index identifier is assigned to each of the element subsets; wherein the index identifier includes a start index and an end index; Mapping the reference element set, the index identifier and the element subset, and storing the mapping relationship; A plurality of the task element sets are generated based on a plurality of the element subsets, and the mapping relationship is marked based on the generated task element sets.
5. A spatial data processing method according to claim 1, characterized in that: The method of concurrently analyzing the data streams of the plurality of subtasks through the NiFi processor cluster, and concurrently processing the data in the spatial database based on the instructions corresponding to the plurality of subtasks, specifically includes: The KafkaConsumer processor in the nifi processor cluster monitors the messages in the distributed message queue system, and when a new message is monitored, the new message is passed to the SdeToShpProcessor processor; Parsing the new message through the SdeToShpProcessor processor to obtain instructions corresponding to the new message; Based on the instruction corresponding to the new message, determine the corresponding spatial query attribute and spatial query condition; wherein the spatial query condition includes a spatial range and a spatial relationship; Constructing a spatial query statement based on the spatial query attribute and the spatial query condition, so as to query the spatial database through the spatial query statement and obtain corresponding spatial data; The acquired spatial data is processed accordingly, and the processed data is packaged into a required format.
6. A spatial data processing method according to claim 1, characterized in that: The processing data is divided into data, the divided data is encrypted in groups, and the encrypted data is transmitted to the distributed file system, specifically including: dividing the processed data into first reference data and second reference data; Performing a first encryption process on a preset first subkey and the first reference data to obtain third reference data, and performing a second encryption process on a preset second subkey and the second reference data to obtain fourth reference data; Based on the preset first spare data, partially replace the third reference data, and based on the preset second spare data, partially replace the fourth reference data; Performing an XOR process on the replaced third reference data and the replaced fourth reference data to obtain fifth reference data; Partially replacing the fifth reference data based on the preset third standby data, and encrypting the replaced fifth reference data with the preset third subkey to obtain sixth reference data; Partially replacing the sixth reference data based on the preset fourth standby data, performing an XOR process on the replaced fourth reference data and the replaced sixth reference data to obtain seventh reference data, performing a replacement process on the seventh reference data, and performing an encryption process on the replaced seventh reference data with the preset fourth subkey; until the number of subkeys participating in the encryption process is greater than the preset number of subkeys, so as to complete the encryption of the working data; The encrypted data is transferred to the distributed file system.
7. A spatial data processing method according to claim 1, characterized in that: The detecting of the acquired task monitoring data corresponding to the NiFi task specifically includes: Performing topology detection on the data flow transmission process corresponding to the NiFi task; wherein the topology detection includes at least one of hanging point detection, overlap detection and intersection detection; Performing encoding detection on the field corresponding to the nifi task; wherein the encoding detection includes at least one of a value range detection and a dictionary value detection on the field; Perform metadata detection on the fields corresponding to the nifi task; wherein the metadata detection includes at least one of field name detection, field type detection and field missing detection.
8. A spatial data processing device, characterized in that: The device comprises a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute the method according to any one of claims 1 to 7.
9. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions can execute the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Distributed monitoring system and method for NIFI task
CN110569174A
Spatial analysis method and device
CN110597935A
Data processing method, apparatus and device, and medium
CN110673891A
Spatial data conversion method based on distributed architecture
CN112199429A
Terminal big data distribution method and system based on NIFI, and storage medium
CN112883110A
