Distributed digital earth data storage method and system

Abstract

The application belongs to the technical field of data storage, and discloses a distributed digital earth data storage method and system, which comprises the following steps: obtaining digital earth data; preprocessing and cutting the digital earth data to obtain fragmented data; performing space-time coding on the fragmented data to obtain coded fragmented data; constructing a distributed hierarchical storage space and storing the coded fragmented data in the corresponding distributed hierarchical storage space; and constructing an index of the fragmented data in the distributed hierarchical storage space. The storage method can improve the loading speed of the digital earth data. The space-time coding of the fragmented data makes each piece of fragmented data have high space-time correlation, which can improve the speed of continuous space-time range data and further improve the loading speed of the digital earth data. Furthermore, the construction of the distributed hierarchical storage space can reduce the storage cost of the data and improve the storage reliability of the data.

Description

Technical Field

[0001] This invention belongs to the field of data storage technology, specifically relating to a distributed digital earth data storage method and system. Background Technology

[0002] With the rapid development of Earth observation technology, modern digital earth platforms need to process petabytes of multi-source heterogeneous data every day. Taking NASA's EOSDIS system as an example, its cumulative storage has exceeded 80 petabytes, covering high-resolution remote sensing images (0.3m~30m), lidar point clouds, real-time meteorological flow, socio-economic vectors and other multi-dimensional data.

[0003] These Digital Earth data exhibit the following significant characteristics:

[0004] Strong spatiotemporal correlation: Each frame of imagery is accompanied by precise spatiotemporal tags (with microsecond-level accuracy);

[0005] Multi-scale characteristics: Global coverage and localized refined data coexist (e.g., a 10km global temperature field and a 1cm urban real-world model).

[0006] Dynamic update requirements: Disaster monitoring and other scenarios require data updates on a minute-by-minute basis.

[0007] Current digital earth systems employ a centralized storage + relational database architecture (such as Oracle Spatial). However, as data continues to grow, this architecture presents at least the following problems in practical applications:

[0008] Scalability limitations: The storage limit of a single node restricts data growth, and the cost of vertical expansion increases exponentially;

[0009] Inefficient query performance: Spatial queries worldwide often take minutes to respond;

[0010] Insufficient fault tolerance: Hardware failures cause data service interruptions (average recovery time > 24 hours). Summary of the Invention

[0011] The purpose of this invention is to provide a distributed digital earth data storage method and system to solve the problems of limited scalability, low query efficiency, and insufficient fault tolerance in the existing technology.

[0012] To achieve the above objectives, the present invention adopts the following technical solution:

[0013] In a first aspect, the present invention provides a distributed digital earth data storage method, the method comprising:

[0014] Acquiring digital earth data;

[0015] Digital Earth data is preprocessed and segmented to obtain fragmented data;

[0016] Spatiotemporal encoding is performed on the fragmented data to obtain the encoded fragmented data;

[0017] Construct a distributed hierarchical storage space and store the encoded fragmented data in the corresponding distributed hierarchical storage space;

[0018] Construct an index for sharded data in a distributed hierarchical storage space.

[0019] Preferably, the digital earth data is remote sensing image data. The digital earth data is preprocessed and segmented to obtain fragmented data, including:

[0020] Perform integrity verification on remote sensing image data to obtain verified remote sensing images;

[0021] The verified remote sensing image is segmented using a multi-threaded process to obtain multiple sub-images.

[0022] Multiple sub-images are resampled at multiple levels to obtain scaled images of several layers. A sub-image and its corresponding scaled image are used as a piece of data.

[0023] Preferably, the fragmented data is spatiotemporally encoded to obtain encoded fragmented data, including:

[0024] Spatial and temporal standardization processes are performed on the fragmented data to obtain the processed data. Spatial standardization includes coordinate system transformation, precision processing, and range verification. Temporal standardization includes time zone conversion, time format processing, and time granularity processing.

[0025] The processed data is then spatially and temporally encoded sequentially to obtain the encoded fragmented data.

[0026] Preferably, the processed data is sequentially spatially encoded and temporally encoded to obtain encoded fragmented data, including:

[0027] The row and column numbers and scaling levels are extracted from the processed data. The row and column numbers are composed of the latitude and longitude values ​​corresponding to the fragmented data.

[0028] Encode the row and column numbers and scaling levels to obtain spatial encoding values;

[0029] Extract the timestamps corresponding to the processed data and encode the timestamps to obtain time-encoded values;

[0030] By associating the time-coded values ​​with the corresponding spatial-coded values, the encoded fragmented data is obtained.

[0031] Preferably, the distributed hierarchical storage space includes several storage nodes, and the encoded fragmented data is stored in the corresponding distributed hierarchical storage space, including:

[0032] All encoded fragmented data are divided to obtain multiple storage datasets, each of which contains at least one encoded fragmented data.

[0033] Each storage dataset is mapped to its corresponding storage node based on load balancing rules.

[0034] Preferably, the load balancing rule is:

[0035] Multiple mapping nodes are created based on the consistent hashing algorithm, and the load capacity of each mapping node allocated to each storage node is calculated.

[0036] Based on the load capacity allocated to each storage node by each mapping node, an allocation overhead function is constructed;

[0037] Solve the allocation cost function to obtain the optimal number of mapping nodes allocated to each storage node;

[0038] Based on the optimal number of mapping nodes allocated to each storage node, create a mapping relationship between each mapping node and the storage node.

[0039] Preferably, mapping each storage dataset to its corresponding storage node based on load balancing rules includes:

[0040] Extract the spatial encoding value and the number of mapping nodes from the encoded fragmented data;

[0041] Calculate the mapping number of each stored dataset based on the spatial encoding value of the encoded fragmented data;

[0042] Based on the number of mapping nodes and the mapping number of each storage dataset, a consistent hashing algorithm is used to determine the mapping node corresponding to each storage dataset.

[0043] Based on the mapping relationship between each mapping node and the storage node, each storage dataset is assigned to the corresponding storage node.

[0044] Preferably, each storage node includes a hot data storage area, a warm data storage area, and a cold data archiving area, with the encoded fragmented data stored in the hot data storage area.

[0045] Preferably, the method further includes: for any stored dataset, generating an identifier for the stored dataset and a unique identifier corresponding to each encoded fragment of data in the stored dataset;

[0046] Constructing an index for sharded data in a distributed hierarchical storage space includes:

[0047] Construct an index tree, which has several levels and each level has at least one index node;

[0048] The hash value of each encoded data segment is calculated using a preset number of hash functions to obtain the hash sequence of each encoded data segment.

[0049] Determine the index path based on the hash sequence of each encoded data segment;

[0050] An index item is constructed based on the identifier of each stored dataset and the unique identifier corresponding to each encoded fragment of data in that stored dataset;

[0051] Add the index entry to the index path to obtain the index of the sharded data.

[0052] Secondly, the present invention provides a distributed digital earth data storage system for implementing the above-mentioned distributed digital earth data storage method, the system comprising:

[0053] The data acquisition module is used to acquire digital earth data;

[0054] The data processing module is used to preprocess and segment digital earth data to obtain fragmented data;

[0055] The data encoding module is used to perform spatiotemporal encoding on the fragmented data to obtain the encoded fragmented data;

[0056] The data storage module is used to construct a distributed hierarchical storage space and store the encoded fragmented data in the corresponding distributed hierarchical storage space.

[0057] The index creation module is used to build indexes for sharded data in a distributed hierarchical storage space.

[0058] Beneficial effects:

[0059] 1. This invention preprocesses and segments digital earth data to obtain fragmented data, and then stores the fragmented data in a distributed manner. This storage method can improve the loading speed of digital earth data.

[0060] 2. This invention performs spatiotemporal coding on the fragmented data, enabling each fragment to have a high degree of strong spatiotemporal correlation, which can improve the data loading speed of continuous spatiotemporal ranges and further improve the loading speed of digital earth data.

[0061] 3. By constructing a distributed hierarchical storage space, this invention can reduce data storage costs and improve data storage reliability. Attached Figure Description

[0062] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings:

[0063] Figure 1 This is a flowchart of a distributed digital earth data storage method provided in one embodiment of the present invention;

[0064] Figure 2 This is a block diagram of a distributed digital earth data storage system provided in one embodiment of the present invention. Detailed Implementation

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0066] Example 1

[0067] Figure 1 This is a flowchart of a distributed digital earth data storage method provided by one embodiment of the present invention. This method can be executed, but is not limited to, by computer devices with certain computing resources, such as cloud servers, personal computers (PCs, referring to a type of multi-purpose computer suitable for personal use in terms of size, price, and performance; desktops, laptops, mini-laptops, tablets, and ultrabooks all fall under the category of personal computers), smartphones, personal digital assistants (PDAs), or wearable devices. Figure 1 As shown in the figure, this embodiment provides a distributed digital earth data storage method, the method including:

[0068] Step S10: Acquire digital earth data; In this embodiment, digital earth data includes, but is not limited to, geographic data and environmental and resource data monitored by various means such as remote sensing satellites, sensors, and ground observation. Among them, geographic data monitored by remote sensing satellites includes, but is not limited to, high-resolution satellite imagery (such as the China-Brazil Resource Satellite, the Environment-1 satellite, etc.), multispectral and radar data, covering surface vegetation, water bodies, cities, geological structures, etc., with spatial resolution from kilometers to centimeters and temporal resolution from minutes to days; for example, dynamic information such as land use change and glacial melting can be obtained through satellite monitoring.

[0069] Step S20: Preprocess and segment the digital earth data to obtain fragmented data.

[0070] In this embodiment, preprocessing and segmentation are mainly for remote sensing image data and terrain data. The remote sensing image is divided into 512×512 pixel tiles, and the terrain data is divided into 3D Tiles blocks to obtain segmented data.

[0071] For digital earth data that is remote sensing image data, the specific steps for preprocessing and segmenting digital earth data are as follows:

[0072] Step S201: Perform integrity verification on the remote sensing image data to obtain the verified remote sensing image. By verifying the integrity of the remote sensing image data, the quality of the remote sensing image data can be verified, which can effectively avoid the inability to find the complete remote sensing image during the query process due to incomplete remote sensing image data.

[0073] Step S202: Perform multi-threaded segmentation processing on the verified remote sensing image to obtain multiple sub-images; In this embodiment, a multi-threaded segmentation task can be created using a GPU (graphics processing unit) to perform segmentation on the verified remote sensing image, which can effectively improve the segmentation processing efficiency of the remote sensing image.

[0074] Step S203: Perform multi-level resampling on multiple sub-images to obtain several layers of scaled images. Use a sub-image and its corresponding scaled image as a piece of data. In this embodiment, by performing multi-level resampling on each sub-image, several layers can be obtained. From the bottom layer to the top layer, the resolution of the image gradually decreases, while the geographical range represented by the image remains unchanged. This structure allows the map to be loaded and browsed quickly. At this time, the user only needs to load the tiles (i.e., piece data) within the current field of view.

[0075] Step S30: Perform spatiotemporal coding on the fragmented data to obtain the coded fragmented data.

[0076] In this embodiment, spatiotemporal coding is performed on the fragmented data to obtain encoded fragmented data, including:

[0077] Step S301: Perform spatial location standardization and temporal information standardization on the fragmented data to obtain the processed data; wherein, spatial location standardization includes: coordinate system transformation, precision processing and range verification; temporal information standardization includes: time zone conversion, time format processing and time granularity processing.

[0078] Among them, coordinate system transformation is used to convert the coordinate system of the pieced data to the WGS84 coordinate system; precision processing is used to retain the effective decimal places of the coordinate values, for example: 116.407526 → 116.4075; range verification is used to verify the validity of the coordinates, such as checking the boundaries of the coordinates.

[0079] The time zone conversion is used to unify the timestamps of the fragmented data to the UTC time zone, for example: 2025-04-15T08:30+08:00 → 2025-04-15T00:30Z; the time format processing is used to unify the time format, adopting the ISO 8601 standard, for example: YYYY-MM-DDTHH:MM:SSZ; the time granularity processing is used to truncate the time according to user needs, for example: 2025-04-15T08:30:45 → 2025-04-15T08:30.

[0080] Step S301: Perform spatial encoding and temporal encoding on the processed data in sequence to obtain the encoded fragmented data.

[0081] In this embodiment, the processed data is sequentially spatially encoded and temporally encoded to obtain encoded fragmented data, including:

[0082] Step S301.1: Extract the row and column numbers and scaling levels from the processed data. The row and column numbers are composed of the latitude and longitude values ​​corresponding to the fragmented data.

[0083] Step S301.2: Encode the row and column numbers and scaling levels to obtain spatial encoding values; the specific steps for encoding the row and column numbers and scaling levels in this embodiment are as follows:

[0084] For each data segment containing three parameters: row number, column number, and level, perform string conversion on the row number, column number, and level to obtain the row value, column value, and level value, and construct an array D:

[0085] ;

[0086] In the formula, The code is in hexadecimal, i = 1, 2, 3, 4;

[0087] Shift the row number left by 16 bits and perform a bitwise OR operation with d5, then perform a bitwise AND operation with d5 ​​to fill the row number into the highest 16 bits of the spatial code value. Shift the column number left by 16 bits and perform a bitwise OR operation with d4, then perform a bitwise AND operation with d4 to fill the row number into the middle 16 bits of the spatial code value. Shift the level left by 58 bits and perform a bitwise OR operation with d1, then perform a bitwise AND operation with d1 to fill the row number into the lowest 58 bits of the spatial code value. Then combine the filled row number, column number, and level to obtain the final spatial code value.

[0088] This embodiment uses bitwise operations to effectively embed rows, columns, and levels into spatial encoding values, thereby realizing the conversion from three-dimensional coordinates to a single encoding value. Furthermore, this method ensures the continuity of the encoding values ​​of fragmented data at the same scaling level, avoiding repetition.

[0089] Step S301.3: Extract the timestamp corresponding to the processed data and encode the timestamp to obtain the time-encoded value.

[0090] Step S301.4: Associate the time encoding value with the corresponding spatial encoding value to obtain the encoded fragmented data.

[0091] In this embodiment, a structure or class can be created to store the final spatial encoding value and temporal encoding value. This external data structure can be used in the application to store and retrieve 3D coordinates and time information.

[0092] Step S40: Construct a distributed hierarchical storage space and store the encoded fragmented data in the corresponding distributed hierarchical storage space.

[0093] In this embodiment, each storage node includes a hot data storage area, a warm data storage area, and a cold data archiving area. The encoded fragmented data is stored in the hot data storage area. The hot data storage area is mainly used to store fragmented data, which is accessed frequently. Therefore, the fragmented data is written to a high-performance storage layer. The warm data storage area is mainly used for intermediate data generated during data processing (such as monthly analysis results) and data with relatively low access frequency (such as data that has not been accessed for 0.5 to 2 years). The hot data storage area is used to store raw image data and data with low access frequency (such as data that has not been accessed for more than 2 years).

[0094] This embodiment achieves an optimal balance between cost, performance, and reliability by storing data in layers, thus offering advantages such as low storage cost, fast access speed, high data reliability, and high intelligence.

[0095] In this embodiment, the encoded fragmented data is stored in the corresponding distributed hierarchical storage space, including:

[0096] Step S401: Divide all encoded fragmented data into multiple storage datasets, each of which contains at least one encoded fragmented data. In this embodiment, multiple fragmented data with consecutive spatial encoding values ​​are divided into the same storage dataset. The number of storage datasets can be determined according to the system configuration, which includes, but is not limited to, the number of storage nodes, the hardware configuration of each storage node (e.g., CPU, memory, disk IOPS, etc.), and the load capacity index of the storage nodes.

[0097] Step S402: Map each storage dataset to the corresponding storage node based on the load balancing rules.

[0098] The load balancing rule is as follows:

[0099] Step S1: Create multiple mapping nodes based on the consistent hashing algorithm and calculate the load capacity of each mapping node allocated to each storage node. The core idea of ​​the consistent hashing algorithm in this embodiment is to allocate data items (such as storage datasets) to virtual nodes on the hash ring, and use the virtual nodes as mapping nodes. Each mapping node corresponds to one or more storage nodes.

[0100] The hash ring is a virtual, continuous ring. Data items are mapped to the hash ring through a hash function and occupy one or more positions. The position of a data item on the ring is calculated based on its hash value, which is unique and uniform. Therefore, each virtual node has a unique mapping position on the hash ring, and multiple virtual nodes can be mapped to the same actual storage node.

[0101] Step S2: Construct an allocation overhead function based on the load capacity of each mapping node allocated to each storage node.

[0102] Step S3: Solve the allocation cost function to obtain the optimal number of mapping nodes allocated to each storage node; the algorithm for solving the allocation cost function can be either particle swarm optimization or genetic algorithm.

[0103] Step S4: Based on the optimal number of mapping nodes allocated to each storage node, create the mapping relationship between each mapping node and the storage node.

[0104] Therefore, based on load balancing rules, each storage dataset is mapped to its corresponding storage node, including:

[0105] Step S401: Extract the spatial encoding value and the number of mapping nodes of the encoded fragmented data;

[0106] Step S402: Calculate the mapping number of each storage dataset based on the spatial encoding value of the encoded fragmented data. In this embodiment, each storage dataset contains multiple encoded fragmented data. The spatial encoding values ​​corresponding to the multiple encoded fragmented data in each storage dataset are accumulated and then divided by the number of encoded fragmented data in the storage dataset. The result is used as the mapping number of the storage dataset.

[0107] Step S403: Based on the number of mapping nodes and the mapping number of each storage dataset, a consistent hashing algorithm is used to determine the mapping node corresponding to each storage dataset. In this embodiment, the number of mapping nodes and the mapping number of each storage dataset are used as inputs to the consistent hashing algorithm. The consistent hashing algorithm can calculate a hash value, and the calculated hash value is used to search for virtual nodes to obtain the mapping node corresponding to each storage dataset.

[0108] The steps for finding virtual nodes using the calculated hash value are as follows: convert the hash value into an angle value on the hash ring (e.g., using modulo operation), start from the beginning of the hash ring (usually 0), and search counterclockwise for the first virtual node with an angle value greater than or equal to the angle value. If no virtual node with an angle value greater than or equal to the angle value is found, the search continues until the first virtual node with an angle value greater than or equal to the angle value is found.

[0109] Step S404: Based on the mapping relationship between each mapping node and the storage node, assign each storage dataset to the corresponding storage node.

[0110] The virtual nodes found in step S403 are mapped to the mapping numbers of the stored datasets. Since step S4 established the mapping relationship between each mapping node and the storage node, each dataset can now be mapped to the corresponding storage node, thus allocating each stored dataset to its corresponding storage node. This embodiment uses a consistent hashing algorithm to calculate the corresponding virtual node based on the number of mapping nodes and the mapping number of each stored dataset. This effectively distributes datasets to storage nodes, achieving load balancing and scalability.

[0111] Step S50: Construct an index for the sharded data in the distributed hierarchical storage space.

[0112] In this embodiment, in order to facilitate users in querying the fragmented data stored in the distributed hierarchical storage space, it is necessary to establish an index for the fragmented data, i.e., an index path, and to realize the query of the fragmented data through the index path.

[0113] As a further optimization of this embodiment, the method further includes: for any stored dataset, generating an identifier for the stored dataset and a unique identifier corresponding to each encoded fragment of the stored data.

[0114] Therefore, constructing an index for sharded data in a distributed hierarchical storage space includes:

[0115] Step S501: Construct an index tree, which has several layers, each layer having at least one index node. In this embodiment, the number of layers of the index tree is determined by the amount of data in the sharded data. The larger the amount of data, the more layers the index tree has. In this embodiment, each layer of the index tree corresponds to a prime number. The larger the number of layers, the larger the value of the prime number. Therefore, the index tree corresponds to a prime number sequence. For example, if the index tree has 5 layers, the prime number sequence is {2,3,5,7,11}. The number of child nodes deployed on each node of the index tree is the same as the value of the prime number corresponding to that layer. For example, the first layer has one node as the root node, and the root node has 2 (the prime number of the first layer) child nodes. Each node of the second layer is connected to 3 (the prime number of the second layer) child nodes.

[0116] Step S502: Calculate the hash value of each encoded fragment based on a preset number of hash functions to obtain the hash sequence of each encoded fragment.

[0117] In this embodiment, the preset number of hash functions is determined by the number of data shards and the number of levels in the index tree. The formula for calculating the preset number is as follows:

[0118] ;

[0119] In the formula, For the preset quantity, Let N be a logarithmic function to the base 2, where N is the number of data fragments and M is the total number of levels in the index tree.

[0120] In this embodiment, the row number, column number, and level number of the fragmented data are input into the hash function, and a hash value can be calculated, resulting in a hash sequence.

[0121] Step S503: Determine the index path based on the hash sequence of each encoded fragment of data.

[0122] In this embodiment, by dividing the hash value in the hash sequence by the total number of levels in the index tree and taking the remainder, a remainder sequence can be obtained. Zeros and duplicate values ​​are removed from the remainder sequence to obtain a new remainder sequence. Each element in the new remainder sequence corresponds to a level in the index tree. For example, if the first remainder is "2", it corresponds to the second level of the index tree. The prime number corresponding to the second level is "3", then "3" is used as the starting point of the index. Assuming the new remainder sequence is {2,3,4}, the index path is {3,5,7}.

[0123] In the second layer and above, each layer has multiple nodes. The fragmented data needs to be mapped to one of these nodes. This is done by dividing the spatial encoding value of the encoded fragmented data by the prime number corresponding to the previous layer and taking the remainder. The corresponding node is then selected based on the remainder. For example, in the second layer: if the result of dividing the spatial encoding value of the encoded fragmented data by the prime number "2" corresponding to the first layer and taking the remainder is "0", then it is mapped to the first node of the second layer; if the result of dividing the spatial encoding value of the encoded fragmented data by the prime number "2" corresponding to the first layer and taking the remainder is "1", then it is mapped to the second node of the second layer.

[0124] Step S504: Construct an index item based on the identifier of each stored dataset and the unique identifier corresponding to each encoded fragment of the stored data. In this embodiment, the index item includes the identifier, the unique identifier, and the hash value corresponding to each encoded fragment.

[0125] Step S505: Add index entries to the index path to obtain the index of the sharded data. Add an index entry to each node on the index path determined in step S503.

[0126] Therefore, when users perform queries, they can quickly find the index path and the corresponding node by retrieving the index item, and then quickly find the corresponding shard data.

[0127] This invention preprocesses and segments digital earth data to obtain fragmented data, and then stores the fragmented data in a distributed manner. This storage method can improve the loading speed of digital earth data. Furthermore, by performing spatiotemporal encoding on the fragmented data, each fragmented data has a high degree of strong spatiotemporal correlation, which can improve the data loading speed of continuous spatiotemporal ranges, further improving the loading speed of digital earth data. Secondly, by constructing a distributed hierarchical storage space, the storage cost of data can be reduced, and the storage reliability of data can be improved.

[0128] Example 2

[0129] Figure 2 This is a block diagram of a distributed digital earth data storage system provided in one embodiment of the present invention. Figure 2 As shown, this embodiment provides a distributed digital earth data storage system for implementing the distributed digital earth data storage method in Embodiment 1. The system includes:

[0130] The data acquisition module is used to acquire digital earth data;

[0131] The data processing module is used to preprocess and segment digital earth data to obtain fragmented data;

[0132] The data encoding module is used to perform spatiotemporal encoding on the fragmented data to obtain the encoded fragmented data;

[0133] The data storage module is used to construct a distributed hierarchical storage space and store the encoded fragmented data in the corresponding distributed hierarchical storage space.

[0134] The index creation module is used to build indexes for sharded data in a distributed hierarchical storage space.

[0135] This embodiment also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the distributed digital earth data storage method in Embodiment 1.

[0136] This embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the distributed digital earth data storage method in Embodiment 1.

[0137] This invention preprocesses and segments digital earth data to obtain fragmented data, and then stores the fragmented data in a distributed manner. This storage method can improve the loading speed of digital earth data. Furthermore, by performing spatiotemporal encoding on the fragmented data, each fragmented data has a high degree of strong spatiotemporal correlation, which can improve the data loading speed of continuous spatiotemporal ranges, further improving the loading speed of digital earth data. Secondly, by constructing a distributed hierarchical storage space, the storage cost of data can be reduced, and the storage reliability of data can be improved.

[0138] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0139] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0140] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A distributed digital earth data storage method, characterized in that, The method comprises: acquiring digital earth data; preprocessing and cutting the digital earth data to obtain fragmented data; spatiotemporal encoding the fragmented data to obtain encoded fragmented data; constructing a distributed hierarchical storage space and storing the encoded fragmented data in the corresponding distributed hierarchical storage space; constructing an index of the fragmented data in the distributed hierarchical storage space; The digital earth data is remote sensing image data, and preprocessing and cutting the digital earth data to obtain fragmented data comprises: performing integrity checking on the remote sensing image data to obtain checked remote sensing images; performing multi-thread segmentation processing on the checked remote sensing images to obtain a plurality of sub-images; performing multi-level resampling on the plurality of sub-images to obtain a plurality of layers of scaled images, taking a sub-image and a corresponding scaled image as a piece of fragmented data; spatiotemporal encoding the fragmented data to obtain encoded fragmented data comprises: performing spatial position standardization processing and time information standardization processing on the fragmented data to obtain processed data; wherein the spatial position standardization processing comprises coordinate system conversion, precision processing and range checking; the time information standardization processing comprises time zone conversion, time format processing and time granularity processing; sequentially performing spatial encoding and time encoding on the processed data to obtain encoded fragmented data, comprising: extracting the row and column numbers and the scaling level from the processed data, wherein the row and column numbers are composed of the latitude value and the longitude value corresponding to the fragmented data; encoding the row and column numbers and the scaling level to obtain a spatial encoding value; The specific steps of encoding the row and column numbers and the scaling level are as follows: For each piece of fragmented data containing row number, column number and level, perform string conversion on the row number, column number and level to obtain row value, column value and level value, and construct an array D: wherein are hexadecimal codes, i = 1, 2, 3, 4, 5; left shift the row number by 16 bits and perform bitwise logical operation with d5 to fill the row number into the highest 16 bits of the spatial encoding value; left shift the column number by 16 bits and perform bitwise logical operation with d4 to fill the column number into the middle 16 bits of the spatial encoding value; left shift the level by 58 bits and perform bitwise logical operation with d1 to fill the level into the lowest 58 bits of the spatial encoding value, then combine the filled row number, column number and level to obtain the final spatial encoding value; extracting the timestamp corresponding to the processed data and encoding the timestamp to obtain a time encoding value; associating the time encoding value with the corresponding spatial encoding value to obtain the encoded fragmented data; The distributed hierarchical storage space comprises a plurality of storage nodes, and storing the encoded fragmented data in the corresponding distributed hierarchical storage space comprises: dividing all the encoded fragmented data to obtain a plurality of storage data sets, each of which contains at least one encoded fragmented data; mapping each storage data set to a corresponding storage node based on a load balancing rule; The load balancing rule is: creating a plurality of mapping nodes based on a consistent hash algorithm and calculating the load capacity of each mapping node allocated to each storage node; constructing a distribution overhead function based on the load capacity of each mapping node allocated to each storage node; Solving the allocation cost function, the optimal number of mapping nodes allocated by each storage node is obtained; According to the optimal number of mapping nodes allocated by each storage node, the mapping relationship between each mapping node and the storage node is created.

2. The distributed digital earth data storage method of claim 1, wherein, Based on the load balancing rule, each storage data set is mapped to the corresponding storage node, including: Extracting the spatial encoding value of the encoded shard data and the number of mapping nodes; According to the spatial encoding value of the encoded shard data, the mapping number of each storage data set is calculated; Based on the number of mapping nodes and the mapping number of each storage data set, a consistent hash algorithm is used to determine the corresponding mapping node of each storage data set; Based on the mapping relationship between each mapping node and the storage node, each storage data set is allocated to the corresponding storage node.

3. The distributed digital earth data storage method of any one of claims 1-2, wherein, Each storage node includes a hot data storage area, a warm data storage area, and a cold data archiving area, and the encoded shard data is stored in the hot data storage area.

4. The distributed Digital Earth data storage method of claim 1, wherein, The method further comprises: for any storage data set, generating an identifier of the storage data set and a unique identifier corresponding to each encoded shard data in the storage data; Building an index of the shard data in the distributed hierarchical storage space, including: Building an index tree, the index tree has several layers, and each layer has at least one index node; Based on a preset number of hash functions, the hash value of each encoded shard data is calculated to obtain the hash sequence of each encoded shard data; Based on the hash sequence of each encoded shard data, the index path is determined; Based on the identifier of each storage data set and the unique identifier corresponding to each encoded shard data in the storage data, an index item is constructed; Adding the index item to the index path to obtain the index of the shard data.

5. A distributed digital earth data storage system for implementing the distributed digital earth data storage method of any one of claims 1-3, characterized in that, The system comprises: A data acquisition module for acquiring digital earth data; A data processing module for preprocessing and cutting the digital earth data to obtain shard data; A data encoding module for spatio-temporal encoding of the shard data to obtain encoded shard data; A data storage module for building a distributed hierarchical storage space and storing the encoded shard data in the corresponding distributed hierarchical storage space; An index creation module for building an index of the shard data in the distributed hierarchical storage space.

Images