A method, device, computer equipment and storage medium for constructing a spatiotemporal index

By generating spatiotemporal grid indexes and prepartitioning in distributed databases, combined with coprocessor equalization division, the problems of low retrieval efficiency and uneven index storage in the existing technology are solved, and efficient spatiotemporal index construction and Query performance improvement.

CN113946700BActive Publication Date: 2024-11-19CHINA UNIV OF GEOSCIENCES (WUHAN)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111042088.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-07
Publication Date
2024-11-19
Estimated Expiration
2041-09-07

AI Technical Summary

Technical Problem

When building spatiotemporal indexes, the prior art fails to effectively combine the time and spatial information of remote sensing data, resulting in low retrieval efficiency and uneven index storage, which affects query performance.

Method used

By obtaining the metadata of remote sensing data, extracting and coding, generating spatiotemporal grid indexes based on time and space attributes, and pre-partitioning in a distributed database, using coprocessors to balance partitioning to ensure the correlation between time and space and Uniformity.

Benefits of technology

It improves the retrieval efficiency of remote sensing data, reduces query time, avoids data skew, and improves system performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113946700B_ABST
    Figure CN113946700B_ABST
Patent Text Reader

Abstract

The present invention provides a method for constructing a spatiotemporal index, including: acquiring remote sensing data, extracting metadata of the remote sensing data, uniformly encoding the metadata, and obtaining first data; encoding the first data based on the time attribute and space attribute of the first data, and obtaining a spatiotemporal grid index; constructing a distributed database, and pre-partitioning a data table in the distributed database based on the spatiotemporal grid index to obtain a plurality of partitions; storing the spatiotemporal grid index in the partitions corresponding to the distributed database, effectively combining the time and space information of the remote sensing data, and improving the retrieval efficiency. Pre-partitioning the data table can avoid resource consumption caused by partition splitting, and ensure that each partition has a balanced amount of data as much as possible.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing data retrieval, and in particular to a method, device, computer equipment and storage medium for constructing a spatiotemporal index. Background Art

[0002] With the development of observation technology, more and more remote sensing satellites are involved in earth observation, making the scale of remote sensing image data larger and larger. In order to retrieve massive remote sensing data more accurately, there are currently multiple indexing methods. At present, there is a method of implementing secondary indexes by building two tables, which can effectively support range queries, but it is difficult to maintain the database when two tables need to be built, and the query efficiency is low when faced with multiple query needs; there is also a method of building a two-layer spatiotemporal index GRIST based on GeoHash and R-Tree. This method uses a two-layer index structure. The first layer uses spatial grid GeoHash encoding, and the second layer uses R-Tree to process time. This index is much better than GeoMesa and PostGIS systems, but it does not effectively combine the combined time information with the spatial information, and the index is stored on the data node, which has the disadvantage of uneven data. When used, the efficiency of retrieval and query is not high. Summary of the invention

[0003] The problem solved by the invention is how to improve the retrieval efficiency of remote sensing data with uneven distribution in time and space.

[0004] To solve the above problems, the present invention provides a method for constructing a spatiotemporal index, comprising:

[0005] Acquire remote sensing data, extract metadata of the remote sensing data, uniformly encode the metadata, and obtain first data; encode the first data based on the time attribute and space attribute of the first data to obtain a spatiotemporal grid index; build a distributed database, pre-partition the data table in the distributed database based on the spatiotemporal grid index to obtain multiple partitions; store the spatiotemporal grid index in the partition corresponding to the distributed database.

[0006] Compared with the prior art, the present invention combines the temporal attribute and spatial attribute of remote sensing data, fully utilizes the temporal and spatial correlation of remote sensing data to encode remote sensing data, thereby improving retrieval efficiency. The data table is pre-partitioned according to the spatiotemporal grid index. Based on the data characteristics of the remote sensing data to be stored, a more appropriate pre-partition mode can be selected to establish a distributed database. Partitioning the spatiotemporal grid index is beneficial to subsequent retrieval and improves retrieval efficiency.

[0007] Optionally, encoding the first data based on the time attribute and the space attribute of the first data to obtain the spatiotemporal grid index includes:

[0008] The first data is globally divided into grid codes to obtain spatial grid codes; the time attributes are recursively subdivided to obtain time codes, wherein the recursive subdivision includes: setting an initial time point and a maximum time span, dividing the maximum time span into preset levels, and for each level of the time span after the division, distinguishing between encoding the time attributes whose time points fall in the first half of the time span and the time attributes whose time points fall in the second half of the time span; combining the spatial grid code with the time code to obtain a spatiotemporal code, and determining the spatiotemporal grid index based on the spatiotemporal code.

[0009] Therefore, the time attribute is recursively subdivided and converted into a one-dimensional code. The time attribute and the space attribute can be represented as a one-dimensional code in combination with the space grid code and stored in the distributed database with the code as the primary key.

[0010] Optionally, the spatiotemporal grid index further includes:

[0011] Satellite sensor code, wherein the satellite sensor code comprises: a six-bit binary code combining the satellite name and type, and the type of the sensor.

[0012] Therefore, adding satellite sensors into the space-time grid index can filter the sources of satellite sensors during retrieval to ensure retrieval efficiency; binary coding can be unified with spatial grid coding and time coding in terms of radix, and six-bit coding can also meet the current number of sensor types.

[0013] Optionally, the building of a distributed database, pre-partitioning a data table in the distributed database based on the spatiotemporal grid index, and obtaining a plurality of partitions comprises:

[0014] Extract the space-time code; combine the space-time code according to the Morton code to obtain the Morton code; convert the Morton code into a preset base to obtain a conversion code; pre-partition the data table in the distributed database according to the base number of the conversion code.

[0015] Therefore, the space-time codes are combined according to the Morton codes to ensure the combination of time and space, and the conversion codes are pre-partitioned to ensure that the number of pre-partitions matches the number of codes, and the most appropriate partitioning mode is selected for pre-partitioning.

[0016] Optionally, after pre-partitioning the data table in the distributed database according to the conversion-coded base number, the method further includes:

[0017] A Coprocessor interface is added to the distributed database to load a coprocessor; and the coprocessor is used to evenly divide the spatiotemporal grid index.

[0018] Therefore, remote sensing data has temporal and spatial inhomogeneity, and a coprocessor is used to evenly divide the spatiotemporal grid index to prevent data skew in the spatiotemporal grid index.

[0019] Optionally, the using the coprocessor to evenly divide the spatiotemporal grid index comprises:

[0020] Obtain the number of spatiotemporal grid indexes in each of the partitions; determine whether the partition is a small partition, wherein the small partition is the partition whose number is less than or equal to a first preset threshold; if so, merge the small partitions in a first preset manner, wherein the first preset manner includes obtaining the conversion code in each small partition, and merging at least two of the small partitions whose absolute value of the difference between the conversion codes is the smallest into one partition.

[0021] Therefore, when the amount of data stored in a partition is small, two small partitions with smaller conversion coding differences are merged into one partition, which can ensure that the spatial and temporal proximity of the data in the partition is not affected as much as possible after the partitions are merged.

[0022] Optionally, the using the coprocessor to evenly divide the spatiotemporal grid index comprises:

[0023] Determine whether the partition is a large partition, wherein the partition in which the number of spatiotemporal grid indexes is greater than or equal to a second preset threshold is taken as the large partition; if so, divide the large partition in a second preset manner, wherein the second preset manner includes splitting the large partition into at least two partitions, and the number of spatiotemporal grid indexes in each partition obtained by the split is less than the second preset threshold and greater than the first preset threshold.

[0024] Therefore, when the amount of data stored in a partition is large, a large partition is split into multiple partitions, and the amount of data in each partition is ensured not to be too much or too little.

[0025] On the other hand, the present invention also provides a spatiotemporal index construction device, comprising:

[0026] An acquisition module is used to acquire remote sensing data, extract metadata of the remote sensing data, uniformly encode the metadata, and obtain first data; an encoding module is used to encode all the first data based on time attributes and space attributes to obtain a spatiotemporal grid index; a construction module is used to construct a distributed database, pre-partition the data table in the distributed database based on the spatiotemporal grid index to obtain multiple partitions; a storage module is used to store the spatiotemporal grid index in the partition corresponding to the distributed database.

[0027] The advantages of the spatiotemporal index construction device over the prior art are the same as those of the spatiotemporal index construction method, which will not be described in detail here.

[0028] The present invention also proposes a computer device, comprising a computer-readable storage medium storing a computer program and a processor, wherein when the computer program is read and executed by the processor, the spatiotemporal index construction method as described above is implemented.

[0029] The advantages of the computer device over the prior art are the same as those of the spatiotemporal index construction method, which will not be described in detail here.

[0030] The present invention also proposes a computer storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is read and executed by a processor, the spatiotemporal index construction method as described above is implemented.

[0031] The advantages of the computer storage medium over the prior art are the same as those of the spatiotemporal index construction method, which will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of a process of constructing a spatiotemporal index according to an embodiment of the present invention;

[0033] Figure 2 This is a schematic diagram of a detailed process of step S200 of the spatiotemporal index construction method according to an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of a detailed process of step S300 of the spatiotemporal index construction method according to an embodiment of the present invention;

[0035] Figure 4 Another flowchart diagram of the spatiotemporal index construction method after step S300 is refined according to an embodiment of the present invention;

[0036] Figure 5 It is another flowchart diagram of the spatiotemporal index construction method after the step S300 is refined according to the embodiment of the present invention;

[0037] Figure 6A flowchart of a method for constructing a spatiotemporal index according to an embodiment of the present invention;

[0038] Figure 7 This is a comparison chart of spatiotemporal query efficiency using different models according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0040] For multi-source remote sensing data, there is an uneven distribution in time and space. The main reason is that after the rapid development of the remote sensing field, more and more remote sensing satellites are involved in earth observation. The orbital parameters of remote sensing data from different satellite sensors are different, and the temporal resolution and spatial resolution are also different. This leads to differences in the storage format, projection standard, resolution, and data update cycle of remote sensing data. As a result, a relatively complete index has not been established in the existing technology to effectively combine remote sensing data with uneven temporal and spatial distribution. Establishing an effective index in the temporal and spatial dimensions can effectively store and analyze massive temporal and spatial data, and speed up retrieval and query efficiency. At this stage, the commonly used remote sensing data organization models do not fully utilize the spatiotemporal information of remote sensing data for spatiotemporal index modeling, resulting in low efficiency in retrieval and access of large-scale multi-source remote sensing data.

[0041] In addition, due to the different satellite orbit parameter designs and different service groups, the remote sensing data is unevenly distributed in the time and space dimensions. For example, the satellite remote sensing data of land satellites that detect earth resources and environment has obvious spatial heterogeneity, which is mainly concentrated in land areas and less distributed in ocean areas. When building indexes for these unevenly distributed data, it is easy to store a large amount of index data on the same node. When a large number of clients access one or several nodes in the cluster, the data read and write requests are too large, which will cause the node to be overloaded, reduce system performance, and affect data access efficiency.

[0042] In order to solve the above problems, the present application proposes a method for constructing an uneven spatiotemporal index in a distributed database, including:

[0043] Step S100, as Figure 1 and Figure 6 As shown, remote sensing data is acquired, metadata of the remote sensing data is extracted, and the metadata is uniformly encoded to obtain first data.

[0044] In one embodiment, remote sensing data includes satellites for earth observation, including land satellites, meteorological satellites, and ocean satellites. Land satellites are mainly used for surveying and monitoring land resources. They are the main type of satellites and have the characteristics of obtaining a wide range of information, observing many and fine objects, and requiring high spatial resolution and spectral resolution. Meteorological satellites are artificial earth satellites that conduct meteorological observations of the earth and the atmosphere from space. They are mainly used for weather forecasts and climate forecasts. Meteorological satellites have the characteristics of a wide observation range, a large number of observations, high temporal resolution, and high quality of observation data. Ocean satellites are satellites that observe the surface of the earth's oceans. They are mainly used for dynamic monitoring of ocean temperature fields, ocean currents, waves, sea salt, etc., and have high requirements for spectral resolution.

[0045] In another embodiment, remote sensing data acquired by Landsat series satellites is used as an example. There are more than 8 million metadata of remote sensing images in the Landsat series, covering the entire world. The data formats are not completely consistent. It is necessary to unify the metadata formats of remote sensing data acquired by different Landsat satellites, including selecting common fields and extended fields to construct a unified metadata format. The unified metadata standard structure model is shown in the following table.

[0046]

[0047] By effectively describing and illustrating the metadata of remote sensing data, the first data can be obtained, and the metadata can be organized into semi-structured or structured data, that is, the metadata can be written into the first data to achieve retrieval, positioning and acquisition of the first data.

[0048] Step S200: encoding the first data based on the time attribute and the space attribute of the first data to obtain a spatiotemporal grid index.

[0049] In one embodiment, an HBase database is used to store data, and the spatiotemporal grid index is the RowKey in HBase.

[0050] Optionally, encoding the first data further includes: encoding a satellite sensor, wherein the encoding structure includes: combining the satellite name and type, and the type of the sensor and performing six-bit binary encoding processing.

[0051] HBase is a distributed, column-oriented storage system built on top of HDFS. HBase can be used when real-time read and write and random access to large-scale data sets are required. It can be expanded by adding nodes from the lower island in a linear manner, with column-oriented storage and permission control, and independent retrieval of columns; HBase tables can be designed to be very sparse; data are all strings, without involving types.

[0052] Unlike data in other fields, remote sensing data has attributes such as time, space, and satellite sensors. Among them, both time attributes and space attributes have certain continuity. For example, when searching and querying remote sensing data, remote sensing data before and after a certain time point is often continuously retrieved, or the data in the surrounding space of the data to be queried is queried together. Therefore, when encoding remote sensing data, it is necessary to consider the correlation between time and space. Accordingly, it is also necessary to associate and store time and space data, which can effectively increase the query efficiency of remote sensing data. In addition, remote sensing data also has satellite sensor attribute fields. Using the attribute fields of satellite sensors to search and query remote sensing data is also a common data retrieval method on remote sensing satellite platforms. Therefore, when building an index, satellite sensors need to be considered as HBase's RowKey.

[0053] In one embodiment, the Landsat1 satellite and MSS sensor are coded as 000000, and have 64 types of addable satellite sensor platforms. The coding is scalable and can ensure that it can adapt to the current number of satellite sensors.

[0054] In the design of HBase table, in addition to RowKey, the design of column cluster is also included. In one embodiment, the specific column cluster of HBase is set to 1, which is fixed to "MD". The columns under the column cluster are ID, Platform, Sensor, ullon, ul lat, ur lon, ur lat, l lon, ll lat, Ir lon, Ir lat, center lon, center lat, startTime, endTime, Level, Path, Format, Cloud, etc. Among them, the four-point longitude and latitude coordinates of remote sensing data are upper left (ul_lon, ul_lat), upper right (ur_lon, ur_lat), lower left (ll_lon, ll_lat), lower right (lr_lon, lr_lat), and the time range is (startTime, endTime). Since it is necessary to obtain detailed information of remote sensing data, it is necessary to store all this information in the column cluster of the HBase table.

[0055] Step S300: construct a distributed database, and pre-partition a data table in the distributed database based on the spatiotemporal grid index to obtain a plurality of partitions.

[0056] The spatiotemporal grid index is associated with remote sensing data to form a grid index based on time and space, and the remote sensing data can be retrieved through the grid index.

[0057] In HBase, RowKey is the primary key used to retrieve records. There are three ways to access rows in HBase:

[0058] 1) Access through a single RowKey.

[0059] 2) Through the range of RowKey.

[0060] 3) Full table scan.

[0061] In one embodiment, the distributed database can be constructed for the remote sensing data and its RowKey only after the construction of the RowKey is completed.

[0062] In one embodiment, the time attribute and the space attribute of the remote sensing data are combined into a one-dimensional RowKey. Through the RowKey, the data table can be pre-partitioned more appropriately, which can reduce the resource consumption caused by the partition splitting of HBase and improve the response speed of HBase.

[0063] When an HBase table is just created, it has only one partition. When the partition is too large and exceeds the preset partition threshold, the table will be split into two partitions. When the table is split, a large amount of resources will be consumed. Frequent partitioning will have a huge impact on the performance of HBase. Therefore, it is necessary to pre-partition the table when it is just created to improve the performance of HBase.

[0064] In one embodiment, a dimensionality reduction operation is performed on spatial attributes and temporal attributes to reduce the three-dimensional time and spatial longitude and latitude information to a one-dimensional space-time grid code to ensure the temporal and spatial relevance of remote sensing data. According to the space-time grid code, the time and space codes generated by different subdivision levels are used as the basis for pre-partitioning of HBase. When the time code and space code are determined, the space code is converted into binary and the binary codes are combined to form a string of binary code combinations. The current number of code combinations is determined, the number of HBase pre-partitions is determined, and then the range of each pre-partition is determined based on the number of pre-partitions and the value of RowKey.

[0065] Step S400: storing the spatiotemporal grid index in a partition corresponding to the distributed database.

[0066] The spatiotemporal grid index and its corresponding remote sensing data are stored in the distributed database to complete the construction of the uneven spatiotemporal index of the distributed database.

[0067] Alternatively, if Figure 3As shown, the construction of the distributed database, pre-partitioning the data table in the distributed database based on the spatiotemporal grid index, and obtaining multiple partitions includes:

[0068] Step S301, extracting each of the spatiotemporal codes.

[0069] In one embodiment, the spatial grid code and the temporal code are one-dimensional codes of spatial attributes and temporal attributes in remote sensing data. The spatial grid code and temporal code in the remote sensing data are extracted, and the space-time code is converted into binary to obtain binary spatial grid code and time code.

[0070] Step S302: Combine each of the space-time codes according to the Morton code to obtain the Morton code corresponding to the space-time code.

[0071] The binary space grid code and the time code in step S301 are transformed and combined according to the Morton code, and all possible combinations of the Morton code are listed to obtain the maximum number of combinations.

[0072] Step S303, convert each of the Morton codes into a preset base to obtain a converted code.

[0073] According to the number of possible coding combinations listed in step S302, the Morton code is converted into a preset base, wherein the base number of the preset base is consistent with the maximum number of combinations.

[0074] Step S304: pre-partition the data table in the distributed database according to the conversion-coded base number.

[0075] In one embodiment, Landsat series remote sensing data is used as the stored remote sensing data, a partition level of 2 is adopted, 64 partitions are divided, and the data index is mapped to each of the 64 partitions. This can ensure that the number of pre-partitions matches the remote sensing data, thereby ensuring the performance of HBase.

[0076] The first-layer pre-partitioning in step S304 is described in detail by taking an example. In one embodiment, a global subdivision grid code is used as the code of the spatial attribute to obtain the spatial grid code; the time attribute is subdivided to obtain the time code. When it is determined that the level of the time code and the spatial code is 1, the corresponding spatial grid code is 0, 1, 2, 3; the time code is 0 or 1. The space code is converted into binary, and the binary results obtained are 00, 01, 10, 11. In this example, the odd bits of the binary result from left to right represent longitude, and the even bits represent latitude. The time code is combined with the longitude and latitude code to produce eight cases: 000, 001, 010, 011, 100, 101, 110, and 111. That is, when the division level is 1, there are eight corresponding combinations of space-time area codes. The code is converted into octal to obtain the conversion code, which is 0, 1, 2, 3, 4, 5, 6, and 7. The length of the code can be shortened. The total number of partitions is 9. Assuming the upper and lower bounds of RowKey are m and M, the RowKey ranges of each partition are [m, 0), [0, 1), [1, 2), [2, 3), [3, 4), [4, 5), [5, 6), [6, 7), [7, M] respectively. When data is stored in the database, the RowKey is distributed to different partitions according to the size. For example, when the beginning of the RowKey code is 01, it will hit the [0, 1) partition, and the remote sensing image will be stored in this partition.

[0077] Similar to the case of level 1, when the level of temporal coding and spatial coding is determined to be 2, the number of partitions is 65 (8 2 +1).

[0078] Alternatively, if Figure 2 As shown, step S200 includes:

[0079] Step S201, performing global grid coding on the first data to obtain spatial grid coding.

[0080] The core idea of ​​the global grid code, GeoSOT code, is based on the principle of earth division. By dividing the earth's surface into specifications, it seeks to build a special grid suitable for spatial information or data organization. The grid system is divided into 10 levels from 0 to 9 using a quadtree grid system with equal longitudes. The 0th level grid is centered at the intersection of the prime meridian and the equator. The earth is transformed into a plane through a simple projection, and the size of the plane is expanded from 180°*360° in earth space to 512°*512°. The first level grid divides the plane into four equal parts based on the 0th level, and each grid size is 256°*256°; the second level continues to divide the plane of the first level into four equal parts based on the first level, and divides them in turn to form different subdivision levels. These grid spaces form subdivision patches with different longitude and latitude ranges. The grid units of each level are encoded by geographic codes according to the anti-Z space filling curve. Therefore, the geographic code is unique. Each code corresponds to a rectangular area. Through dimensionality reduction, the two-dimensional longitude and latitude information is represented as a one-dimensional spatial code. Through this encoding method, the codes of the same area or similar areas can have the same prefix, which improves the spatial correlation between remote sensing data. When the level is higher, that is, the data is divided into smaller and smaller, the spatial area of ​​each subdivision patch can be represented by the longitude and latitude of its center point. Therefore, when calculating its GeoSOT code, only the GeoSOT code corresponding to the longitude and latitude of its center point is needed to represent the space of the area, converting the two-dimensional longitude and latitude information into a one-dimensional code, and improving the spatial proximity of remote sensing data, effectively improving the retrieval efficiency of data.

[0081] Step S202, recursively subdivide the time attributes to obtain time coding, the recursive subdivision includes: setting an initial time point and a maximum time span, dividing the maximum time span into preset levels, and for each level of the time span after the division, distinguishing between encoding the time attributes whose time points fall in the first half of the time span and the time attributes whose time points fall in the second half of the time span.

[0082] In one embodiment, encoding time includes setting 1970 as the initial time point and the maximum time span as 128 years, which means that the time range is from 1970 to 2098. The code for 0-64 years is set to 0, and the code for time between 64-128 years is set to 1, and recursive subdivision is performed in sequence to form different subdivision levels. When subdivided to the 7th level, each code represents a time span of 1 year, and 1 year is expanded to 16 months; when subdivided to the 11th level, each code represents a time span of 1 month, and 1 month is expanded to 32 days. Based on time coding, 1970-2098 can be subdivided into 21 levels, each code represents a time span of 1 hour, and the subdivision levels of time coding are as shown in the following table. Since 21 levels are listed, the lower right corner of the table is blank.

[0083]

[0084] Step S203: Combine the spatial grid code with the time code to obtain a spatiotemporal code, and determine the spatiotemporal grid index based on the spatiotemporal code.

[0085] In one embodiment, in order to clearly indicate the code after combining the spatial grid code with the time code, the combined code is named GeoSOT-ST code. The GeoSOT-ST code includes: a six-bit binary code.

[0086] In another embodiment, the spatiotemporal grid index includes a combination of GeoSOT-ST code, satellite sensor code and remote sensing data ID. For example, the spatiotemporal grid index of a remote sensing data is 100011-001001-122, indicating that the GeoSOT-ST code of the remote sensing data is 100011, the satellite sensor code is 001001, the ID of the remote sensing data is 122, and the remote sensing data is stored in the corresponding partition. The time attribute is additionally processed, and the acquisition time of the remote sensing data, such as the long type value of year, month, day, hour, minute and second, is stored in the table as a timestamp to increase the time relevance of the remote sensing data.

[0087] In the spatiotemporal grid index, the GeoSOT-ST coding part is the main field for indexing remote sensing data; the satellite sensor coding is the screening of satellites and sensors; the timestamp is the screening of the precise time range of remote sensing data; and the ID ensures the uniqueness of the spatiotemporal grid index.

[0088] Optionally, after pre-partitioning the data table in the distributed database according to the conversion-coded base number, the method further includes:

[0089] A Coprocessor interface is added to the distributed database to load the coprocessor.

[0090] The coprocessor is used to evenly divide the spatiotemporal grid index.

[0091] Because remote sensing data is unevenly distributed in time and space, the RowKey of remote sensing data will also have different numbers of differences, resulting in different data volumes in each partition. After the data is stored in the database, the coprocessor is used to further merge and split the partitions to ensure that the data volume in each partition is roughly balanced, avoid data skew, and ensure system performance.

[0092] Alternatively, if Figure 4 As shown, using the coprocessor to evenly divide the spatiotemporal grid index includes:

[0093] Step S310, obtaining the number of spatiotemporal grid indexes in each of the partitions.

[0094] Step S311, determining whether the partition is a small partition, wherein the small partition is the partition whose number is less than or equal to a first preset threshold.

[0095] Step S312: If yes, merge the small partitions in a first preset manner, wherein the first preset manner includes obtaining the conversion code in each small partition, and merging at least two of the small partitions with the smallest absolute value of the conversion code difference into one partition.

[0096] Optionally, refer to Figure 5 As shown, the using the coprocessor to evenly divide the spatiotemporal grid index includes:

[0097] Step S320, obtaining the number of spatiotemporal grid indexes in each of the partitions.

[0098] Step S321, determining whether the partition is a large partition, wherein the partition whose number of spatiotemporal grid indexes is greater than or equal to a second preset threshold is regarded as the large partition.

[0099] Step S322: If yes, divide the large partition in a second preset manner, wherein the second preset manner includes splitting the large partition into at least two partitions, and the number of spatiotemporal grid indexes in each partition obtained by the split is less than a second preset threshold and greater than the first preset threshold.

[0100] Since remote sensing data is unevenly distributed in time and space, if only pre-partitioning is performed, some areas will store only a small amount of remote sensing data, while some areas will concentrate a large amount of remote sensing data. Therefore, under the premise of pre-partitioning, the HBase coprocessor is used to further merge and split the unevenly distributed partitions on the server side. If the amount of data in a partition is less than or equal to the first preset threshold, it means that the amount of data in this partition is too small and needs to be merged with other partitions; if the amount of data in a partition is greater than or equal to the second preset threshold, it means that the amount of data in this partition is too large and needs to be split into multiple partitions.

[0101] When merging or splitting partitions, the temporal and spatial proximity of remote sensing data may be affected, so the temporal and spatial correlation of remote sensing data needs to be taken into account. When merging partitions, they need to be merged according to the temporal and spatial correlation, such as merging the partitions corresponding to the data groups with similar conversion codes into one partition; when there is too much data in a partition, the sub-partition needs to be split, and part of the data can be transferred to the adjacent partition by expanding the conversion code range of the adjacent partitions before and after / or reducing the conversion code range of the partition; when the data volume of the partition before and after this partition is large, the partition is split into more partitions to ensure data balance and prevent data skew. After multiple merging and splitting operations, the index of remote sensing data can be evenly distributed in different partitions.

[0102] like Figure 7 As shown in the figure, the retrieval efficiency of various models is tested through experiments. Three comparative models are used in the experiment: (1) Lon-Lat-Time longitude and latitude, time storage model (2) GeoSOT-Time model, spatial filtering using GeoSOT coding, combined with time filtering; (3) GeoSOT-ST non-partitioned model, using the default HBase partitioning strategy. All metadata are stored in the database, and points within the map center point range are randomly selected, and then query areas of 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the entire map are constructed, and then the precision and query efficiency of different query ranges are tested.

[0103]

[0104] After multiple experiments, the average value was calculated, and the exact amount of query data and the amount of query data of the comparison model were calculated. It was found that the precision of the Lon-Lat-Time model was 100%, the precision of GeoSOT-Time was 89%, and the precision of GeoSOT-ST was 96%. Figure 7 As shown, the solution GeoSOT-ST-2 of the present invention has a significant improvement in precision and query efficiency compared with other solutions.

[0105] A spatiotemporal index construction device according to another embodiment of the present invention includes:

[0106] An acquisition module is used to acquire remote sensing data, extract metadata of the remote sensing data, uniformly encode the metadata, and obtain first data; an encoding module is used to encode the first data based on time attributes and space attributes to obtain a spatiotemporal grid index; a construction module is used to construct a distributed database, pre-partition a data table in the distributed database based on the spatiotemporal grid index to obtain multiple partitions; a storage module is used to store the spatiotemporal grid index in the partition corresponding to the distributed database.

[0107] The advantages of the spatiotemporal index construction device over the prior art are the same as those of the spatiotemporal index construction method, which will not be described in detail here.

[0108] A computer device according to another embodiment of the present invention includes a computer-readable storage medium storing a computer program and a processor. When the computer program is read and executed by the processor, the spatiotemporal index construction method described above is implemented.

[0109] The advantages of the computer device over the prior art are the same as those of the spatiotemporal index construction method, which will not be described in detail here.

[0110] A computer storage medium according to another embodiment of the present invention stores a computer program. When the computer program is read and executed by a processor, the spatiotemporal index construction method described above is implemented.

[0111] The advantages of the computer storage medium over the prior art are the same as those of the spatiotemporal index construction method, which will not be described in detail here.

[0112] Although the disclosure is as above, the protection scope of the disclosure of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the disclosure, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A method for constructing a spatiotemporal index. It is characterized in that include: Acquire remote sensing data, extract metadata of the remote sensing data, and uniformly encode the metadata to obtain first data; Based on the time attribute and space attribute of the first data, the first data is encoded to obtain a spatiotemporal grid index, including: globally dividing the first data into grids to obtain a spatial grid code; recursively subdividing the time attribute to obtain a time code, wherein the recursive subdivision includes: setting an initial time point and a maximum time span, dividing the maximum time span into preset levels, and for each level of the time span after the division, distinguishing between encoding a time attribute whose time point falls in the first half of the time span and a time attribute whose time point falls in the second half of the time span; combining the spatial grid code with the time code to obtain a spatiotemporal code, and determining the spatiotemporal grid index based on the spatiotemporal code; Constructing a distributed database, pre-partitioning a data table in the distributed database based on the spatiotemporal grid index to obtain a plurality of partitions, including: extracting each of the spatiotemporal codes; combining each of the spatiotemporal codes according to the Morton code to obtain a Morton code corresponding to the spatiotemporal code; converting each of the Morton codes into a preset base to obtain a conversion code; pre-partitioning the data table in the distributed database according to the base number of the conversion code; The spatiotemporal grid index is stored in a partition corresponding to the distributed database.

2. The method for constructing a spatiotemporal index according to claim 1, It is characterized in that The spatiotemporal grid index also includes: Satellite sensor code, wherein the satellite sensor code comprises: a six-bit binary code combining the satellite name and type, and the type of the sensor.

3. The method for constructing a spatiotemporal index according to claim 1, It is characterized in that After pre-partitioning the data table in the distributed database according to the conversion-coded base number, the method further includes: Adding a Coprocessor interface to the distributed database and loading a coprocessor; The coprocessor is used to evenly divide the spatiotemporal grid index.

4. The method for constructing a spatiotemporal index according to claim 3, It is characterized in that The using the coprocessor to evenly divide the spatiotemporal grid index comprises: Obtaining the number of spatiotemporal grid indexes in each of the partitions; Determine whether the partition is a small partition, wherein the small partition is the partition whose number is less than or equal to a first preset threshold; If so, the small partitions are merged in a first preset manner, wherein the first preset manner includes obtaining the conversion code in each small partition, and merging at least two of the small partitions with the smallest absolute value of the difference between the conversion codes into one partition.

5. The method for constructing a spatiotemporal index according to claim 3, It is characterized in that The using the coprocessor to evenly divide the spatiotemporal grid index comprises: Determining whether the partition is a large partition, wherein the partition whose number of spatiotemporal grid indexes is greater than or equal to a second preset threshold is regarded as the large partition; If so, the large partition is divided in a second preset manner, wherein the second preset manner includes splitting the large partition into at least two partitions, and the number of spatiotemporal grid indexes in each partition obtained by the split is less than the second preset threshold and greater than the first preset threshold.

6. A spatiotemporal index construction device, It is characterized in that include: An acquisition module, which is used to acquire remote sensing data, extract metadata of the remote sensing data, and uniformly encode the metadata to obtain first data; A coding module, which is used to encode all the first data based on time attributes and space attributes to obtain a spatiotemporal grid index, including: performing global grid coding on the first data to obtain a spatial grid code; recursively subdividing the time attribute to obtain a time code, wherein the recursive subdivision includes: setting an initial time point and a maximum time span, subdividing the maximum time span into preset levels, and for each level of the time span after the subdivision, distinguishing between encoding a time attribute whose time point falls in the first half of the time span and a time attribute whose time point falls in the second half of the time span; combining the spatial grid code with the time code to obtain a spatiotemporal code, and determining the spatiotemporal grid index based on the spatiotemporal code; A construction module, which is used to construct a distributed database, pre-partition the data table in the distributed database based on the spatiotemporal grid index to obtain multiple partitions, including: extracting each of the spatiotemporal codes; combining each of the spatiotemporal codes according to the Morton code to obtain the Morton code corresponding to the spatiotemporal code; converting each of the Morton codes into a preset base to obtain a conversion code; pre-partitioning the data table in the distributed database according to the base number of the conversion code; A storage module is used to store the spatiotemporal grid index in a partition corresponding to the distributed database.

7. A computer device, It is characterized in that It comprises a computer-readable storage medium storing a computer program and a processor, and when the computer program is read and executed by the processor, the spatiotemporal index construction method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is read and executed by a processor, the method for constructing a spatiotemporal index according to any one of claims 1 to 5 is implemented.