Dataset fragmentation method based on two-dimensional geographic position information

Abstract

The invention discloses a dataset fragmentation method based on two-dimensional geographic position information. The method includes the steps that (1) two-dimensional geographic position information of each datum is converted into a binary geoHash value; (2) fragmentation is carried out according to the binary geoHash values, each fragment has a public geoHash prefix, and a fragment index is built or updated in the fragment process; (3) when a new datum is added, the fragment with the longest public geoHash prefix with the new datum is found in the index, then the datum is inserted into the fragment, and if the size of the fragment succeeds a preset value due to insertion of the datum, the fragment is fragmented according to the step (2). According to the method, the two-dimensional geographic position information is converted into the geoHash values for data fragmentation, so that it is guaranteed that data which are adjacent in geographic position are fragmented on the same fragment as much as possible, and a good optimization function is achieved on distributed application based on geographic positions.

Description

technical field [0001] The invention belongs to the technical field of data storage, and in particular relates to a data set fragmentation method based on two-dimensional geographic location information. Background technique [0002] In the era of big data, when the amount of data reaches the level of megabytes, a single memory or disk cannot store it. In this case, the dataset needs to be stored in a distributed manner. When performing distributed storage, the data set needs to be fragmented, so as to facilitate the organization, management, migration and other operations of data in units of slices. When the current popular NOSQL database MongoDB performs distributed storage, it first fragments the data set, and then manages the fragments on multiple servers. [0003] When we design a database, we usually divide a large global database according to its data items (ie fields) or according to certain characteristics of a keyword. Here we call it data fragmentation, and the ...

AI Technical Summary

Problems solved by technology

[0008] In the current sharding schemes, a one-dimensional attribute is generally selected as the shard key, such as the ID of the data, and then this dimensional attribute is directly divided into different slices according to the linear range. These sharding methods cannot guarantee the same The data on the slice is geographically similar, or even if GeoHash is used to shard, it only uses GeoHash as a common one-dimensional attribute, and then divides it according to the range of GeoHash, which does not make good use of the characteristics of GeoHash and cannot be very Good guarantee that on-chip data is geographically contiguous

Images