GeoSOT Grid Data Calculation Method

Abstract

The present invention provides a GeoSOT grid data calculation method, comprising: reading corresponding data in a GeoSOT grid data calculation unit in response to a user request; a cluster control node judging whether to start a calculation task corresponding to the user request based on the computing power and usage status of a space-time grid server; when the calculation task is started, an application manager is created on a child node manager for the calculation task, and the application manager starts a data analysis program corresponding to the GeoSOT grid data calculation unit, and allocates the data analysis program to a container of a corresponding space-time grid server to complete the calculation, wherein the container is allocated based on the user's pre-settings and the cluster's workload, and when the container completes the data analysis program, the allocated container is reclaimed. By allocating resources through the cluster control node and allocating response computing power to different regions, global calculation tasks can be completed within the same or similar time, thereby achieving the purpose of accelerating large-scale data parallel computing.

Description

Technical Field

[0001] The present invention belongs to the technical field of spatial grid computing and processing, and in particular relates to a GeoSOT grid data computing method. Background Art

[0002] The complexity of spatiotemporal big data and its analysis algorithms increases the difficulty of implementing algorithms on high-performance computing systems. A distributed computing framework oriented to the spatial characteristics of spatiotemporal big data is an important requirement for high-performance computing of geographic data.

[0003] In existing technologies, 1) there are several issues with the data cube partitioning of Earth Observation Data Cubes (EODCs): since users only receive the final results and not the raw data information, the accuracy of time values may introduce uncertainty in requests and responses. The current OGC WMS Earth Observation profile only uses time parameters for acquisition time. A fundamental problem with the geographic service system (WMS) is the unclear semantics of the parameters. Acquisition time, processing time, and release time are unclear in the temporal dimension. In the spatial dimension, the gridding based on the regional minimum bounding matrix (MBR) results in inconsistent spatial distribution of data for each computational task. 2) Current distributed geographic data computing frameworks, such as SparkGIS and Hadoop-GIS, all organize data by discretizing geographic regions into a format suitable for use in mature high-performance computing frameworks. They lack a unified paradigm that is unique and can be freely scaled within the grid space as a guiding principle. 3) In terms of computing power scheduling strategy, the two-tier scheduling system solution improves the scalability of the system and realizes modularity that is easy to maintain and expand. However, due to the inability to monitor the overall situation of the system, it will encounter performance interference between heterogeneous loads. Although the hybrid scheduler can complement the problems of the scheduling system to a certain extent, it greatly increases the complexity of the system and is not suitable for specific application scenarios of spatiotemporal big data.

[0004] Different from universal big data, spatiotemporal big data, as a special data form in the field of geographical science, follows the following two spatial principles:

[0005] (1) Spatial autocorrelation. Spatial autocorrelation, also known as spatial dependence, is an expression of the first law of geography. It refers to the interdependence between spatial samples (variables); in contrast to correlation, "auto" specifically refers to the correlation between single-class samples. Spatial autocorrelation statistics are used to measure a basic property of geographic data: the degree of interdependence between data at a certain location and data at other locations. Due to the influence of spatial interaction and spatial diffusion, geographic data may no longer be independent of each other, but correlated. The so-called spatial autocorrelation is to study the spatial autocorrelation between a spatial unit and its surrounding units in space, based on a certain characteristic value, through statistical methods, to analyze the characteristics of the spatial distribution phenomenon of these spatial units. Global and local autocorrelation indices can be used to measure the degree of aggregation or dispersion of spatial feature attribute values. Different autocorrelation coefficients are suitable for different data types. Spatial weight is the premise and basis for spatial autocorrelation analysis.

[0006] The first law of geography states that everything is related to everything else, but things that are close together are more correlated than things that are far away. Spatial autocorrelation imposes requirements on the system deployment of geographic data servers. First, data storage location. Spatial data access and analysis are often based on a specific spatial range. However, many existing big data platforms (such as Hadoop) use a shared-nothing storage architecture, organized in blocks. This requires large amounts of data to be transferred across the network during geographic data processing, impacting data response efficiency between services and clients. In distributed data computing, data locality can reduce data movement (network transmission or I / O), but this is not readily apparent with traditional shared disk system architectures. Second, in the context of ultra-large-scale spatiotemporal data processing, significant computing power must be globally distributed. For example, Geoweb's cell server organization assigns servers to grid regions. Compared to the Earth Simulator's design of assigning threads to grid regions, this achieves geographically localized data aggregation. This solution considers a similar design.

[0007] (2) Spatial heterogeneity - the second law of geography. Spatial heterogeneity refers to the unevenness and complexity of the spatial distribution of ecological processes and patterns, which can be understood as the sum of spatial patchiness and gradients. Spatial heterogeneity in geography refers to the spatial unevenness (quality) of the distribution of spatial sample patterns and evolutionary processes. The isolation of things in geographic space creates differences (i.e., heterogeneity). In the field of geographic data analysis, the intuitive phenomenon is the difference in data volume, data complexity, and query access frequency in different regions. Under the strategy of "average distribution" of computing power (such as assigning a thread to the yin-yang grid for concurrent processing), it is difficult to achieve balanced processing time for tasks, which restricts computing efficiency to a certain extent.

[0008] Existing geographic data systems rarely consider computing power allocation based on spatial heterogeneity in their computing power allocation strategies. Most universal big data processing platforms address data concurrency through load balancing mechanisms driven by data volume. Therefore, in addition to grid-based computing power organization, regional scalability is also required to meet the data needs of each region. Summary of the Invention

[0009] In response to the problems existing in the prior art, the present invention provides a GeoSOT grid data calculation method, which at least partially solves the problem of slow large-scale data parallel calculation speed existing in the prior art.

[0010] The present disclosure provides a method for calculating GeoSOT grid data, including:

[0011] In response to user requests, read the corresponding data in the GeoSOT grid data calculation unit;

[0012] The cluster's control node determines whether to start the computing task corresponding to the user request based on the computing power and usage status of the space-time grid server;

[0013] When a computing task is started, an application manager is created on the child node manager for the computing task, and the application manager starts the data analysis program corresponding to the GeoSOT grid data computing unit, and allocates the data analysis program to the container of the corresponding space-time grid server to complete the calculation. The container is allocated based on the user's pre-settings and the cluster's workload. When the container completes the data analysis program, the allocated container will be reclaimed.

[0014] Optionally, read the corresponding data in the GeoSOT grid data calculation unit, including:

[0015] Obtain the data type and time and space range requested by the user;

[0016] Convert the data type and spatiotemporal range into a grid code, match the grid code with the data block that matches the primary key and time range, and then obtain the grid computing unit information of the corresponding data type within the requested range;

[0017] Based on the grid computing unit information, the grid addressing service transmits the corresponding data request to the space-time grid server of the physical storage;

[0018] Based on the grid code and time information of the request information, the multidimensional array in the data block is read in the corresponding space-time grid server as the spatial data of the grid computing unit, and the request code is the primary key of the grid computing unit.

[0019] Optionally, before the step of converting the data type and spatiotemporal range into grid code, matching the grid code with data blocks that conform to the primary key and time range, and then obtaining the grid computing unit information of the corresponding data type within the requested range, it includes constructing a GeoSOT grid data computing unit model, the GeoSOT grid data computing unit model includes a grid computing unit spatiotemporal primary key model and a grid computing unit data partitioning model, and the grid computing unit data partitioning model is the spatial data of the grid computing unit.

[0020] Optionally, the primary key in the space-time primary key model of the grid computing unit is used to identify the encoding of data and add the space-time attributes of the data in the computing cycle. The primary key of the grid computing unit includes a logical data model.

[0021] Optionally, the primary key is used as the code for identifying the data and is expanded to include latitude, longitude and elevation codes.

[0022] Optionally, extended with latitude, longitude, and elevation encoding, including:

[0023] Calculate the longitude and latitude codes according to the conversion formula between GeoSOT code and longitude and latitude to obtain the longitude and latitude codes;

[0024] The latitude and longitude codes are interleaved bit by bit to obtain a binary one-dimensional code.

[0025] Optionally, the conversion formula is:

[0026]

[0027]

[0028] in, is the longitude code, B n is the longitude, n is the coding level, is the latitude code, L n is the latitude.

[0029] Optionally, the spatial data of the grid computing unit is constructed based on raster data.

[0030] Optionally, spatial data of grid computing units is constructed based on raster data, including:

[0031] Traverse all grid points to calculate the grid set covering the grid surface;

[0032] Based on the grid set, the grid surface is split at the grid boundary;

[0033] Traverse the grid surfaces obtained by segmentation and allocate them to the space-time grid server of the space they belong to through data addressing storage strategy.

[0034] Optionally, before traversing all grid points to calculate the grid set covering the grid surface, the following is also included:

[0035] Calculate the minimum bounding rectangle of the grid surface and locate the coordinates of the four corner points of the grid surface;

[0036] Determine the data range scale, which is represented by the grid level.

[0037] The GeoSOT grid data calculation method provided by the present invention allocates resources through the control nodes of the cluster and allocates response computing power to different areas, which can achieve the completion of global computing tasks in the same or similar time, thereby achieving the purpose of accelerating large-scale data parallel computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0039] Figure 1 A flowchart of a GeoSOT grid data calculation method provided in an embodiment of the present disclosure;

[0040] Figure 2 A schematic diagram of an organizational model provided in an embodiment of the present disclosure;

[0041] Figure 3 A schematic diagram of a server architecture provided for an embodiment of the present disclosure;

[0042] Figure 4 A schematic diagram of a space-time grid server provided in an embodiment of the present disclosure;

[0043] Figure 5 A schematic diagram of the GeoSOT binary two-dimensional cross-coding provided in an embodiment of the present disclosure;

[0044] Figure 6A schematic diagram of weather radar data calculation based on a grid distributed computing framework provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0045] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0046] It should be clear that the following embodiments of the present disclosure are described through specific concrete examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0047] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.

[0048] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0049] Additionally, in the following description, specific details are provided to provide a thorough understanding of the examples. However, one skilled in the art will appreciate that the aspects described can be practiced without these specific details.

[0050] The following problems currently exist in organizing tasks for ultra-large-scale grid computing hardware:

[0051] (1) A geographic server organization architecture based on discrete space division is needed to solve the problem of data localization and aggregation management on the one hand, and to allocate response computing power to different regions on the other hand, so that global computing tasks can be completed in the same or similar time, achieving the goal of large-scale data parallel computing acceleration;

[0052] (2) The current computing power allocation model for earth science data is relatively simple (e.g., the Earth Simulator allocates one thread to each yin-yang grid, Geoweb allocates at least one server to each grid, etc.), which makes it difficult to provide good scalability for spatially differentiated data distribution;

[0053] (3) When processing ultra-large-scale spatiotemporal data tasks, there is no geographic logic and semantic association between data rows, making it difficult to organize data locally according to spatial regions.

[0054] The following problems exist in the current high-performance computing process of spatiotemporal big data:

[0055] (1) A time-space integrated data grid / cube partitioning method is needed to uniformly organize distributed computing units, provide time-space identification for data units from storage-reading-computation-result-association merging, and obtain the location and time of the results through encoding without the user having access to the original data, thus facilitating the tracing and merging of results.

[0056] (2) To address the I / O issues in ultra-large-scale data computing cycles, we draw on the principle of fully distributed decentralized design to design a node management and scheduling model for the Earth's regional divisions, achieving the goal of I / O acceleration.

[0057] (3) In view of the principle of spatial heterogeneity, each regional node server should be able to provide scalability based on the data volume, complexity, access frequency, etc. of its own region.

[0058] like Figure 1 As shown, this embodiment discloses a GeoSOT grid data calculation method, including:

[0059] In response to user requests, read the corresponding data in the GeoSOT grid data calculation unit;

[0060] The cluster's control node determines whether to start the computing task corresponding to the user request based on the computing power and usage status of the space-time grid server;

[0061] When a computing task is started, an application manager is created on the child node manager for the computing task, and the application manager starts the data analysis program corresponding to the GeoSOT grid data computing unit, and allocates the data analysis program to the container of the corresponding space-time grid server to complete the calculation. The container is allocated based on the user's pre-settings and the cluster's workload. When the container completes the data analysis program, the allocated container will be reclaimed.

[0062] Optionally, read the corresponding data in the GeoSOT grid data calculation unit, including:

[0063] Obtain the data type and time and space range requested by the user;

[0064] Convert the data type and spatiotemporal range into a grid code, match the grid code with the data block that matches the primary key and time range, and then obtain the grid computing unit information of the corresponding data type within the requested range;

[0065] Based on the grid computing unit information, the grid addressing service transmits the corresponding data request to the space-time grid server of the physical storage;

[0066] Based on the grid code and time information of the request information, the multidimensional array in the data block is read in the corresponding space-time grid server as the spatial data of the grid computing unit, and the request code is the primary key of the grid computing unit.

[0067] Optionally, before the step of converting the data type and spatiotemporal range into grid code, matching the grid code with data blocks that conform to the primary key and time range, and then obtaining the grid computing unit information of the corresponding data type within the requested range, it includes constructing a GeoSOT grid data computing unit model, the GeoSOT grid data computing unit model includes a grid computing unit spatiotemporal primary key model and a grid computing unit data partitioning model, and the grid computing unit data partitioning model is the spatial data of the grid computing unit.

[0068] Optionally, the primary key in the space-time primary key model of the grid computing unit is used to identify the encoding of data and add the space-time attributes of the data in the computing cycle. The primary key of the grid computing unit includes a logical data model.

[0069] Optionally, the primary key is used as the code for identifying the data and is expanded to include latitude, longitude and elevation codes.

[0070] Optionally, extended with latitude, longitude, and elevation encoding, including:

[0071] Calculate the longitude and latitude codes according to the conversion formula between GeoSOT code and longitude and latitude to obtain the longitude and latitude codes;

[0072] The latitude and longitude codes are interleaved bit by bit to obtain a binary one-dimensional code.

[0073] Optionally, the spatial data of the grid computing unit is constructed based on raster data.

[0074] Optionally, spatial data of grid computing units is constructed based on raster data, including:

[0075] Traverse all grid points to calculate the grid set covering the grid surface;

[0076] Based on the grid set, the grid surface is split at the grid boundary;

[0077] Traverse the grid surfaces obtained by segmentation and allocate them to the space-time grid server of the space they belong to through data addressing storage strategy.

[0078] Optionally, before traversing all grid points to calculate the grid set covering the grid surface, the following is also included:

[0079] Calculate the minimum bounding rectangle of the grid surface and locate the coordinates of the four corner points of the grid surface;

[0080] Determine the data range scale, which is represented by the grid level.

[0081] In a specific example, in the context of computing power organization, data localization, data discretization decomposition, and the fact that ultra-large-scale spatiotemporal data processing cannot be separated from large computing power and storage devices, a large-scale grid computing power organization model is designed for large-scale geographic data processing tasks, using the GeoSOT global segmentation framework as the spatial guidance framework. While ensuring the spatial localization characteristics of the data, it provides high scalability and ultimately forms a hardware organization solution for ultra-large-scale spatiotemporal data computing. The grid server organization model is as follows: Figure 2 shown.

[0082] The grid computing power organization architecture is based on the GeoSOT global subdivision reference grid system, and a spatial grid computing power organization model is established. Through a three-tier server architecture, spatial autocorrelation and spatial heterogeneity are taken into account, and a scalable grid cluster server is allocated to each grid area. The device management of each space-time grid server is carried out through the grid IP mapping model. Finally, a grid server system was developed based on edge computing box devices.

[0083] GeoSOT space-time grid computing power organization model architecture

[0084] Based on the GeoSOT global grid system, a grid computing power organization model for the geographic science data computing platform is proposed. In combination with the edge computing architecture, a three-tier cluster architecture is designed. The top layer is the global root server, i.e., the addressing server. Each grid is assigned a scalable master-slave cluster server. The master server can only display spatial information, while the slave server can display both spatial and temporal information. The global root server manages each grid service in a unified manner through a grid-coded IP mapping model. Within the space-time grid server, the master node uses spatial coding to determine the unique GeoSOT grid IP identity itself and provides regional expansion and online migration functions. The specific contents are as follows:

[0085] GeoSOT grid space-time server model:

[0086] The GeoSOT grid space-time server model (referred to as the space-time server model) is based on geospatial principles and establishes a mapping relationship between real space and computing resources based on the GeoSOT grid subdivision system. In order to describe the system structure in a universal way and use general programming to describe the space-time grid server, the basic abstract type is defined as follows:

[0087] 1) Geospatial Region Grid: The Earth subspace defined by the GeoSOT grid, which is identified in server configuration and data organization, with a hierarchical attribute L∈[0,31];

[0088] 2) Computing nodes in the region: server nodes allocated on the space-time server architecture. Each grid location is assigned a node. The node is responsible for the storage, management, analysis, and computation of the spatiotemporal data belonging to the region. The corresponding grid level of the node is l∈[L,32];

[0089] 3) Server resources in the node: Assets are server resources in the region, which can be associated with storage resources, memory, network, physical servers, cloud servers, edge computing power, etc.

[0090] 4) Map Function: Resource allocation rules, which establishes a set of Node and Asset allocation and management functions in the grid space, which can be understood as a grid server management platform.

[0091] In the space-time server model, each GeoSOT grid area is mapped to a cluster server (Cluster) based on the principle of geographic space, which is called a space-time grid server. The configuration of each space-time grid server is defined by the mapping function MapFuction, which is as follows:

[0092] C:Cluster×N:Node→A:Assets,

[0093] To assign a space-time grid server to a given GeoSOT grid area, we first need to determine the GeoSOT grid level, which determines the degree of discreteness and region size to partition the global space. Table 1 shows the number of global grids corresponding to each GeoSOT grid level. During the initial partitioning, low-level grids are generally selected to define space-time grid servers. Due to GeoSOT's quadtree partitioning rules, a 6-level grid corresponds to a total of 1012 global partitions.

[0094] Table 1. Statistics of GeoSOT grid size and number of grids

[0095]

[0096] GeoSOT Grid Space-Time Server Architecture

[0097] Based on the GeoSOT space-time server model and the first-level GeoSOT grid as the server organization level, a grid space-time server system architecture is proposed, such as Figure 3 As shown in Figure 1, by dividing the world into four grids, four logically corresponding cluster servers are formed, forming a three-tier cluster architecture. Compared to the distributed hardware organization architecture, each space-time server in this system architecture has a clear geographical regional meaning.

[0098] Taking into account the principles of spatial autocorrelation and spatial heterogeneity, since each server is designed as a distributed cluster, it can be expanded according to the different data characteristics and access popularity within the grid area. For example, in a cloud-based high-performance computing environment (such as Alibaba Cloud and other computing services), computing resources can be allocated through dynamic workloads to achieve load balancing. A common way to achieve dynamic load balancing is to add cluster nodes to meet large-scale data or high-density computing tasks.

[0099] Furthermore, the system architecture is based on an edge architecture, using a method that maps GeoSOT grid codes to device IP addresses to communicate and manage the space-time grid server. The space-time grid server corresponds to a specific grid space on Earth, establishes a unique grid code mapping for each device IP address, and provides space-time grid server code resolution capabilities within the global root server.

[0100] Space-time grid server code resolution is a prerequisite for bidirectional data storage, user access and data addressing, and serves as an intermediate condition between the global root server and the space-time grid server.

[0101] Space-time server architecture

[0102] This embodiment designs a prototype system based on domestic edge devices according to the GeoSOT grid space-time server model. The hardware server is divided into two parts, including a global root server and a space-time grid server with a specified number of grids.

[0103] Global root servers:

[0104] The global root server is responsible for the management and maintenance of the entire space-time cluster architecture, including grid server GeoIP addresses, grid server access and unloading, grid server expansion, cluster computing power scheduling and other management services.

[0105] The space-time grid server IP address management service manages each space-time grid server through GeoSOT codes. Based on the edge computing management framework, a mapping table is established between GeoSOT codes and space-time grid server IP addresses. The device code is then parsed to obtain the device's corresponding real-world location, supporting rapid addressing and positioning of data and spatial access requests. For example, in the prototype system cluster management platform, space-time grid servers are defined using a first-level GeoSOT grid code, which divides the Earth's space into four quadrants, corresponding to the northeast, southeast, northwest, and southwest hemispheres.

[0106] The management platform provides the IP address and host name of each space-time grid server, as well as the longitude and latitude range and representative area after GeoSOT grid code analysis. Taking the host kvbox01 as an example, its IP address 172.17.240.185 represents the northeastern hemisphere through grid code mapping. Through device code analysis, it can be seen that this space-time grid server serves the data storage management and computing tasks of the grid space of 0 to 180 degrees longitude and 0 to 90 degrees latitude on the earth.

[0107] Space-time grid server:

[0108] This system uses GeoSOT level 1 coding to map the earth into four grid spaces to organize computing power, and constructs space-time servers through cloud servers and edge computing boxes. Figure 4 Cloud-based servers are available in heterogeneous configurations, either standalone or clustered. Computing resources include threads, memory, and GPUs, while storage resources include SSDs.

[0109] A physical server configuration was built using domestically produced edge computing boxes, with four edge computing boxes corresponding to four GeoSOT Level 1 grid areas. The edge boxes can be clustered using virtual machines or by adding blade servers. The system is scaled with one master and three slave virtual machines per space-time grid server.

[0110] GeoSOT grid data calculation unit model

[0111] Based on the GeoSOT grid computing power organization architecture, this embodiment proposes a memory-based distributed grid elastic computing framework. First, the block data structure improves parallel I / O efficiency. At the same time, a GeoSOT spatiotemporal grid index is established for the data block units, forming a multi-dimensional grid data cube to avoid unnecessary data reading and preprocessing. Then, a computing power scheduling model for geographic grid regions is designed based on the complexity of the computing task. Elastic computing is the dynamic allocation of resources on demand. Due to uneven geographical distribution and different application focus areas, the amount of grid data and computing power will vary in time and space. Storage and computing devices can be decomposed or merged based on the grid as needed to improve overall utilization efficiency and system performance.

[0112] Based on the GeoSOT global grid, a grid system with longitude and latitude and axis parallel is used to perform regional division of spatiotemporal big data.<key,value> The PairRDD computational dataset format was improved, and the GeoSOT grid computation unit was designed. During the data computation cycle, this eliminated the need for preprocessing steps such as local reference grid division, such as the minimum outsourcing region (MBR), for different task requests, accelerating the computation cycle. Furthermore, the spatial and temporal location of data was described using grid encoding semantics using a space-time coding scheme for geographic space and time information. This allowed data to be traced back through the encoded primary key throughout the computation cycle. The final computation results (such as Earth science data visualization) also incorporated geographic locations through space-time coding.

[0113] Relying on the spatial multi-layer nested design of GeoSOT coding, the grid computing unit can determine the grid level according to the amount of data in the area, which can largely ensure the consistency of data scale and improve the efficiency of parallel computing.

[0114] Grid computing unit space-time primary key model:

[0115] Each step in the spatiotemporal big data computation cycle (e.g., archiving, extraction, processing, analysis, and visualization) influences each other, and differences in data formats within the computational steps can make the entire cycle very complex and lengthy. For example, consider the disparity between the data storage model in HDFS and the data model in netCDF data files. When climate data files are stored in HDFS, they are automatically broken down into a set of individual blocks and distributed across the cluster. This file splitting process does not consider file content, making the data in each block difficult for both humans and machines to understand and trace. It is difficult for users to determine which node a block is stored on or which sub-blocks a block contains.

[0116] Based on the spatiotemporal data management method, data storage and addressing can be guided by grid data encoding and storage strategies. However, in the computing cycle, it is difficult to avoid data storage block and data cube scale consistency. In order to incorporate the spatiotemporal attribute primary key design of data in the computing cycle, this embodiment proposes a grid spatiotemporal primary key model, including semantically rich and parseable spatial encoding and data. Let the data computing unit be data cube , logical data model, data calculation unit (data cube) = primary key (GeoSOT code), data value

[0117] Specifically:

[0118] data cube =<key(GeoSOTcode),datavalue>

[0119] GeoSOTcode can be expanded as follows:

[0120]

[0121] in<key,value> In the data set, key is the primary key of the data unit. The dataset primary key is used as a code to identify the data, and can be expanded to include latitude, longitude, and elevation codes. Considering the inclusion relationship of multiple levels of grids, the GeoSOT binary one-dimensional code is used as the primary key of the GeoSOT grid unit.

[0122] The extended binary two-dimensional encoding process is:

[0123] Step 1: Calculate the codes of latitude and longitude respectively according to the conversion formula between GeoSOT code and longitude and latitude:

[0124]

[0125]

[0126] in, is the longitude code, B n is the longitude, n is the coding level, is the latitude code, L n is the latitude.

[0127] Step 2: Cross the longitude and latitude codes to obtain a binary one-dimensional code, such as Figure 5 shown.

[0128] Grid computing unit data partitioning model:

[0129] GeoSOT grid computing unit<key,value> The value in the format is the spatial data within the area to which the grid belongs. Using raster data as an example, we construct the calculation unit data. First, we calculate the minimum bounding rectangle of the raster surface and locate the coordinates of the four corner points. Second, we need to determine the scale of the data range, which we represent by inputting the grid level. Then, we determine the range and location of the data segmentation based on the scale of the grid. The following operations are performed on the input raster:

[0130] 1. Calculate the grid set covering the grid surface by traversing all grid points.

[0131] 2. Based on the grid set, split the grid surface at the grid boundary.

[0132] 3. Traverse the grid surface obtained by segmentation and allocate the grid surface to the grid server of the corresponding space through data addressing storage strategy.

[0133] Grid computing unit read

[0134] The way data units are read is one of the important factors that determine the computation time, which determines the workload balance between the computation nodes within the grid server. The steps to read grid computation units are:

[0135] 1. The user enters the data type and time and space range;

[0136] 2. The system converts the user input into a grid code (i.e., the logical information of the grid computing unit), matches the data blocks that match the spatiotemporal code primary key and time range, and then obtains the grid computing unit information of the corresponding data type within the requested range;

[0137] 3. The grid addressing service passes the corresponding data request to the physical storage grid server. This process relies on the grid storage and management methods in Chapter 3 to achieve data locality and maintain workload balance.

[0138] 4. Based on the request encoding and timing information, the file system reads the multidimensional array within the data block as the value of the grid computation unit, with the request encoding as the computation unit key. Spark manages the memory and hard drive storage levels on each node to accommodate situations where available memory is less than the ingested dataset. Computation units that exceed device memory are automatically spilled to disk as serialized objects.

[0139] GeoSOT grid calculation process:

[0140] When a user submits a computational program to the system, the system first generates the requested data based on the user's request, preparing it until the computational units are fully loaded. After the computational task is submitted, the system's master management node (the master node is the cluster's control node, responsible for the management and control of the entire cluster) begins evaluating the computing power and usage within the grid servers. It then determines whether the memory usage of the computational units should overflow to disk based on the memory usage of the computational units.

[0141] When a compute task is started, the Resource Manager creates an application manager for the task on the child node manager. The child node starts the application manager, which then starts the Spark program corresponding to the compute unit. The Sparkmaster instance is the application driver and is responsible for scheduling tasks to the Spark executors running on the assigned containers.

[0142] On each grid server, the Spark master is the first process to load containers. Containers are dynamically assigned to Spark executors based on requests from the Spark master. These executors are thread pools controlled by the Spark master to execute specific tasks. Because clients can pre-set driver and executor specifications, the cluster management platform will accommodate these requests based on the current workload and lease containers to Spark applications for a specified period of time. After all tasks are completed, the resource manager will revoke all container resources leased to Spark-based applications. From an application perspective, assuming a batch of remote sensing images is to be processed using a grid computing framework, the underlying Spark master will first create grid computing cells from the raw data. It will then map user-defined transformation functions on the cells to parallel tasks to be executed by Spark executors. The Spark master will perform all execution tasks, such as scheduling tasks and managing intermediate results. RDD partitions are optimally cached in memory to speed up the next transformation.

[0143] Take meteorological science data visualization tasks as an example, Figure 6 As shown, Figure 6 The text in the text is for example only and does not limit the technical solution. After the original raster data is organized and divided by the computing unit, each data block will be independently rendered and calculated. These calculation data are called by the grid server computing power stored in the method of this embodiment to perform Spark task calculation. Since the grid computing unit is composed of<key,val ue> The calculation results of each data block are expressed in a form that accompanies the entire cycle. The location information of the GeoSOT grid is included in the calculation results of each data block, so that all results can be aggregated through grid filling and returned to the user's interactive interface for display or next step.

[0144] The method disclosed in this embodiment has the following advantages:

[0145] 1) To address the computational acceleration issues for ultra-large-scale spatiotemporal data, this embodiment provides a distributed elastic computing framework based on the GeoSOT geographic spatial reference grid, which addresses several key challenges to a certain extent: 1. Unified spatiotemporal grid organization of data; 2. An efficient grid-based integrated data model as a computational carrier for multidimensional spatiotemporal raster data; 3. Data decentralized I / O, efficient extraction of localized data, and execution of query, retrieval, analysis, and visualization tasks; 4. Computing power allocation that complies with spatial heterogeneity.

[0146] 2) The GeoSOT grid elastic computing framework ensures data decentralization and localized computing, while also meeting local scalability requirements due to geographic heterogeneity and resolving the uneven data distribution caused by spatial heterogeneity.

[0147] 3) The GeoSOT global gridding framework provides a spatial and temporal integrated data cube partitioning scheme, which can use the time partitioning method as one of the reference coefficients for constructing the data cube and can accurately express the temporal attributes of the data using a unified cubic standard.

[0148] 4) Through the geographic server organization architecture based on the principle of discrete space division, data localization and aggregation management are realized; by allocating response computing power to different regions, global computing tasks can be completed in the same or similar time, achieving the goal of accelerating large-scale data parallel computing.

[0149] 5) Utilize the server organization framework to ensure data locality. In other words, the shorter the distance between data storage and processing, the less data transmission is required during parallel processing.

[0150] 6) The generated response computing power distribution model provides good scalability for spatially differentiated data distribution.

[0151] 7) The model establishes a computing environment for spatially structured data that can reflect the changing state of data in real time and can flexibly provide the ability to implement online services.

[0152] This embodiment discloses a GeoSOT grid data calculation device, including:

[0153] A reading module is used to read corresponding data in the GeoSOT grid data calculation unit in response to a user request;

[0154] A judgment module, used for the cluster's control node to determine whether to start the computing task corresponding to the user request based on the computing power and usage status of the space-time grid server;

[0155] The computing module is used to start creating an application manager on the child node manager for the computing task when the computing task is started. The application manager starts the data analysis program corresponding to the GeoSOT grid data computing unit and allocates the data analysis program to the container of the corresponding space-time grid server to complete the calculation. The container is allocated based on the user's pre-set settings and the cluster's workload. When the container completes the data analysis program, the allocated container will be reclaimed.

[0156] The electronic device disclosed in this embodiment includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may, for example, include read-only memory (ROM), a hard disk, a flash memory, etc.

[0157] The processor may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. In one embodiment of the present disclosure, the processor is configured to execute the computer-readable instructions stored in the memory, causing the electronic device to execute all or part of the steps of the GeoSOT grid data calculation method described in the various embodiments of the present disclosure.

[0158] Those skilled in the art should understand that in order to solve the technical problem of how to obtain a good user experience, this embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the scope of protection of this disclosure.

[0159] An electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In RAM, various programs and data required for the operation of the electronic device are also stored. The processing device, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0160] Typically, the following devices can be connected to the I / O interface: input devices such as sensors or visual information acquisition devices; output devices such as display screens; storage devices such as magnetic tapes and hard disks; and communication devices. The communication devices can allow electronic devices to communicate with other devices (such as edge computing devices) wirelessly or wired to exchange data.

[0161] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, all or part of the steps of the GeoSOT grid data calculation method of the embodiment of the present disclosure are executed.

[0162] For detailed description of this embodiment, please refer to the corresponding description in the aforementioned embodiments, which will not be repeated here.

[0163] According to an embodiment of the present disclosure, a computer-readable storage medium stores non-transitory computer-readable instructions, which, when executed by a processor, execute all or part of the steps of the GeoSOT grid data calculation method of each embodiment of the present disclosure.

[0164] The above-mentioned computer-readable storage media include, but are not limited to, optical storage media (e.g., CD-ROMs and DVDs), magneto-optical storage media (e.g., MOs), magnetic storage media (e.g., magnetic tapes or mobile hard disks), media with built-in rewritable non-volatile memory (e.g., memory cards), and media with built-in ROM (e.g., ROM cartridges).

[0165] For detailed description of this embodiment, please refer to the corresponding description in the aforementioned embodiments, which will not be repeated here.

[0166] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0167] In the present disclosure, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The block diagrams of the devices, devices, equipment, and systems involved in the present disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "including," "comprising," "having," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0168] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0169] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0170] Various changes, substitutions, and modifications may be made to the technology described herein without departing from the teachings defined by the appended claims. Moreover, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of things, means, methods, and actions described above. Currently existing or later developed processes, machines, manufactures, compositions of things, means, methods, or actions that perform substantially the same function or achieve substantially the same results as the corresponding aspects described herein may be utilized. Accordingly, the appended claims include within their scope such processes, machines, manufactures, compositions of things, means, methods, or actions.

[0171] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0172] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A GeoSOT grid data calculation method, characterized in that: include: In response to user requests, read the corresponding data in the GeoSOT grid data calculation unit; The cluster's control node determines whether to start the computing task corresponding to the user request based on the computing power and usage status of the space-time grid server; When a computing task is started, an application manager is created on the child node manager for the computing task. The application manager starts the data analysis program corresponding to the GeoSOT grid data computing unit and allocates the data analysis program to the container of the corresponding space-time grid server to complete the calculation. The container is allocated based on the user's pre-set settings and the cluster's workload. When the container finishes executing the data analysis program, the allocated container will be reclaimed. Read the corresponding data in the GeoSOT grid data calculation unit, including: Obtain the data type and time and space range requested by the user; Convert the data type and spatiotemporal range into a grid code, match the grid code with the data block that matches the primary key and time range, and then obtain the grid computing unit information of the corresponding data type within the requested range; Based on the grid computing unit information, the grid addressing service transmits the corresponding data request to the space-time grid server of the physical storage; Based on the grid code and time information of the request information, the multidimensional array in the data block is read in the corresponding space-time grid server as the spatial data of the grid computing unit, and the request code is the primary key of the grid computing unit; Before the step of converting the data type and spatiotemporal range into a grid code, matching the grid code with a data block that conforms to the primary key and time range, and then obtaining the grid computing unit information of the corresponding data type within the requested range, a GeoSOT grid data computing unit model is constructed. The GeoSOT grid data computing unit model includes a grid computing unit spatiotemporal primary key model and a grid computing unit data segmentation model. The grid computing unit data segmentation model is the spatial data of the grid computing unit. The primary key in the space-time primary key model of the grid computing unit is used to identify the encoding of the data and add the space-time attributes of the data in the computing cycle. The primary key of the grid computing unit includes a logical data model; The primary key is used as the code for identification data, which is expanded to the latitude, longitude and elevation codes; The extension is for latitude, longitude and elevation encoding, including: Calculate the longitude and latitude codes according to the conversion formula between GeoSOT code and longitude and latitude to obtain the longitude and latitude codes; The latitude and longitude codes are interleaved bit by bit to obtain a binary one-dimensional code.

2. The GeoSOT grid data calculation method according to claim 1, characterized in that: The conversion formula is: in, is the longitude code, B n is the longitude, n is the coding level, is the latitude code, L n is the latitude.

3. The GeoSOT grid data calculation method according to claim 1, characterized in that: The spatial data of the grid computing unit is constructed based on raster data.

4. The GeoSOT grid data calculation method according to claim 3, characterized in that: Construct spatial data of grid computing units based on raster data, including: Traverse all grid points to calculate the grid set covering the grid surface; Based on the grid set, the grid surface is split at the grid boundary; Traverse the grid surfaces obtained by segmentation and allocate them to the space-time grid server of the space they belong to through data addressing storage strategy.

5. The GeoSOT grid data calculation method according to claim 4, characterized in that: Before traversing all grid points to calculate the grid set covering the grid surface, it also includes: Calculate the minimum enclosing rectangle of the grid surface and locate the coordinates of the four corner points of the grid surface; Determine the data range scale, which is represented by the grid level.

Images