A Sampling System in Spatiotemporal Data Query Processing

Abstract

The present invention belongs to the field of data analysis technology, and is specifically a sampling system in spatiotemporal data query processing. The system of the present invention includes a spatiotemporal grid divider and a sample generator: the spatiotemporal grid divider divides grids on the spatiotemporal coordinate system according to a certain granularity, and allocates sample capacity to each grid; the specific division granularity can be defined according to actual needs; the sample generator uniformly samples in each grid according to the sample capacity allocated by the divider, and summarizes it into a final sample. The present invention can help analysts obtain more comprehensive samples from spatiotemporal big data, discover information that is easily missed by simple uniform sampling, and increase the reliability of sample analysis conclusions.

Description

Technical Field

[0001] The present invention belongs to the technical field of data analysis, and in particular relates to a data sampling system in data analysis. Background Art

[0002] Nowadays, in order to cope with fierce market competition, enterprises need to conduct a large amount of analysis on their data sets to make wise strategic decisions. Among them, the spatiotemporal data analysis of users has gradually become an important source of information for analyzing user preferences. This is especially true for industries such as social media, catering and entertainment that are closely related to user activity habits.

[0003] For spatiotemporal data that are closely correlated in time and space, if a simple uniform sampling method is used, the distribution characteristics of the data in time and space will be ignored, such as the density of spatial distribution (such as the difference between the city center and the suburbs) and the concentration of temporal distribution (such as the difference between peaks and troughs). If the spatiotemporal correlation between these data is not considered, the sample will not be representative enough, resulting in the accuracy of the approximate query results failing to meet the requirements. Therefore, in order to take this spatiotemporal correlation into account when sampling, we propose a special data sampling method based on spatiotemporal partitioning. Summary of the invention

[0004] The purpose of the present invention is to provide a sampling system in spatiotemporal data query processing to improve the quality of spatiotemporal data sampling and facilitate better mining of effective information in spatiotemporal data sets.

[0005] The sampling system in the spatiotemporal data query processing provided by the present invention includes two submodules: a spatiotemporal grid divider and a sample generator; wherein:

[0006] The spatiotemporal grid divider is responsible for dividing the spatiotemporal coordinate system involved in the spatiotemporal dataset into a series of grids according to the user-defined division granularity and in combination with the metadata information (data distribution, etc.) of offline statistics; then, the spatiotemporal grid divider evenly distributes the sample capacity to each grid; for grids with sparse coordinate points, that is, the total number of coordinate points is less than the average sample capacity, the excess part will be evenly distributed to the remaining grids again to ensure full utilization of the sample capacity.

[0007] The sample generator randomly and uniformly samples the coordinate points in each grid according to the sample capacity allocated by the spatiotemporal grid divider, and calculates the actual sampling rate of each grid; then the sampling results are aggregated into a final sample set, and the predicted value of the statistic required by the user is calculated.

[0008] In the present invention, in the spatiotemporal grid divider, the specific processing flow is as follows:

[0009] (1) Dividing the space-time grid

[0010] According to the grid width w input by the user, for an n-dimensional spatiotemporal dataset Where N is the total number of coordinates, and its boundaries are represented by {[l1,u1],[l2,u2],…,[l n ,u n ],[l n+1 ,u n+1 ]}, where l i ,u i Respectively represent the lower and upper bounds of the i-th dimension, i∈[1,N]. Divide at intervals of w in each dimension:

[0011] {[l i ,l i +w],[l i +w,l i +2w],…,[l i +(k-1)w,l i +kw],[l i +kw,u i ]}; i = 1, ..., n + 1;

[0012] in, To get positive downwards;

[0013] Then the space-time coordinate system is divided into: grids, each grid has p i coordinate points, i≥0, i=1..|G|;

[0014] (2) Allocating sample capacity

[0015] According to the sampling ratio λ specified by the user, the sample capacity can be obtained; S = λN, the i-th grid is allocated samples (i∈[1,N]). Assume that the number of grids in the grid subset G1 is less than s i samples, that is, s i >p i , then the extra s i -p i The samples are evenly distributed to the grids that are not in G1. Repeat this step until G1 is an empty set. Finally, the actual sample capacity distribution is obtained.

[0016] In the present invention, in the spatiotemporal grid divider, the specific processing flow is as follows:

[0017] (1) Perform grid-by-grid sampling. For each grid g i , use random uniform sampling to obtain s′ i Coordinate point S′ iAs a sample, the actual sampling rate of the grid is

[0018] (2) Integrate the results of the previous sampling step to obtain the final sample set. The final sample set is:

[0019] S={S′1,S′2,…,S′ |G|},

[0020] During operation, the approximate query result of the user query on the data set is calculated based on the above sample set:

[0021]

[0022] Where f(·) is the calculation function of the relevant statistics on the sample data; the specific expression of f(·) is defined according to actual needs. For example, to calculate the sample capacity, f(x) = |x| can be taken.

[0023] In the sampling system of the present invention, sampling is performed in the grids divided according to user requirements and database metadata information, ensuring that the grids with sparse data have a sufficient sampling rate.

[0024] The present invention can help spatiotemporal data analysts obtain more comprehensive and representative samples, and avoid the impact of some sparse area samples on further data analysis due to sampling omissions. Experiments show that under the premise of spatiotemporal data sampling with the same sample capacity, the samples obtained by the present invention can better cover the samples in sparse areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The spatiotemporal dataset that the user needs to analyze.

[0026] Figure 2 Samples obtained for uniform sampling.

[0027] Figure 3 Samples obtained for the present invention.

[0028] Figure 4 It is an architecture diagram of the sampling system in the spatiotemporal data query processing of the present invention. DETAILED DESCRIPTION

[0029] The present invention is further described below through specific examples.

[0030] Example: The user wants to perform sampling analysis on a certain part of the spatiotemporal data in the database ( Figure 1 ), but the samples obtained by simple uniform sampling are difficult to reflect some characteristics of the data set, and often miss some coordinate points in sparse areas ( Figure 2). Using this system, users can specify the granularity of spatiotemporal data division, such as 1° latitude and longitude intervals. The system will grid the data set according to the granularity provided by the user, allocate sample capacity and sample each grid one by one, and finally obtain a comprehensive and representative sample ( Figure 3 ). The samples collected using this system can cover samples in sparse areas, ensuring the quality of sampling. It helps users obtain better analysis results without missing potential effective information. The entire process of using the present invention is as follows Figure 4 shown.

Claims

1. A sampling system in spatiotemporal data query processing, characterized in that: It includes two submodules: space-time grid divider and sample generator; among them: (1) The spatiotemporal grid divider submodule divides the spatiotemporal coordinate system involved in the spatiotemporal dataset into a series of grids according to the user-defined division granularity and the metadata information of offline statistics; then, the sample capacity is allocated to each grid according to a certain rule. The specific processing flow of this module is as follows: (1.1) Dividing the space-time grid According to the grid width w input by the user, for an n-dimensional spatiotemporal dataset Where N is the total number of coordinates, and its boundaries are represented by {[l1,u1],[l2,u2],…,[l n ,u n ],[l n+1 ,u n+1 ]}, where l i ,u i Respectively represent the lower and upper bounds of the i-th dimension, i∈[1,N]; each dimension is divided into intervals of w: {[l i ,l i +w],[l i +w,l i +2w],…,[l i +(k-1)w,l i +kw],[l i +kw,u i ]};i=1,…,n+1; in, To round down; Then the space-time coordinate system is divided into: grids, each grid has p i coordinate points, i≥0, i=1..|G|; (1.2) Allocating sample capacity According to the sampling ratio λ specified by the user, the total sample capacity is obtained; S = λN, the i-th grid is allocated samples, i∈[1,N]; assuming that the number of grids in the grid subset G1 is less than s i samples, that is, s i >p i , then the extra s i -p i The samples are evenly distributed to the grids that are not in G1; repeat this step until G1 is an empty set; finally, the actual sample capacity distribution is obtained (2) The sample generator submodule randomly and uniformly samples the coordinate points in each grid according to the sample capacity allocated by the spatiotemporal grid divider, and calculates the actual sampling rate of each grid; then, the sampling results are aggregated into a final sample set and the predicted value of the statistic is calculated; the specific processing flow of this module is as follows: (2.1) Perform grid-by-grid sampling, for each grid g i , use random uniform sampling to obtain s i ′ Coordinate point S i ′ As a sample, the actual sampling rate of the grid is (2.2) Integrate the sampling results of the previous step to obtain the final sample set. The final sample set is: S={S1 ′ ,S2 ′ ,…,S | ′ G| }, During operation, the approximate query result of the user query on the data set is calculated based on the above sample set: Where f(·) is the calculation function of the relevant statistics on the sample data; the specific expression of f(·) is defined according to actual needs.

2. The sampling system according to claim 1, characterized in that: Sampling is performed in the grid divided by the system according to user requirements and database metadata information to ensure that the grid with sparse data has sufficient sampling rate.

Images