A load-adaptive distributed spatial text stream data processing system

Abstract

The present invention provides a load-adaptive distributed spatial text stream data processing system, comprising a data input layer, which is configured to parse and pre-process spatial text stream data; a partition routing layer, which is configured to index and route spatial text stream data to relevant spatial partitions; a matching processing layer, which is configured to store and match spatial text stream data that meets the conditions of spatial query; a load balancing layer, which is configured to monitor the load of each processing unit in real time and dynamically adjust the load; and a data storage layer, which is configured to store outdated spatial text stream data. The present invention can be used for distributed processing of spatial text stream data, etc.

Description

Technical Field

[0001] The present invention belongs to the field of stream data processing, and in particular relates to a load-adaptive distributed spatial text stream data processing system. Background Art

[0002] The development of location-based services has led to an explosive growth in spatial text data. Social media users use social platforms such as Weibo and WeChat to post posts containing location information. Life service platforms such as Meituan provide local services such as food delivery and errands. Taxi-hailing apps also issue orders to nearby taxi drivers based on customer location information. While these services are convenient for the public, they also lead to the generation of large-scale geo-tagged text data streams. These text data containing spatial information contain a lot of valuable information, so a spatial text stream data processing system with real-time and high throughput is particularly important.

[0003] Existing technologies usually use frameworks such as Hadoop or Spark to process large-scale data sets. SpatialHadoop and Hadoop-GIS, which support parallel processing of spatial text data by extending Hadoop, are disk-based spatial analysis systems, which are not good enough for the real-time performance of streaming spatial text data. GeoSpark and LocationSpark are distributed spatial data analysis systems based on Spark. The former uses two spatial index structures, quadtree and R-tree, to support spatial links, range queries, and kNN queries; the latter extends Spark through query schedulers and local query executors. These disk- and memory-based systems focus on static spatial data, are not suitable for large-scale streaming data, and have huge communication costs and load imbalance problems. Summary of the invention

[0004] The purpose of the present invention is to provide a load-adaptive distributed spatial text stream data processing system to address the defects of the prior art in that it is not suitable for spatial text stream data, and has high communication costs and load imbalance. The system can be used for distributed processing of spatial text stream data and can support more real-time, efficient and load-balanced query work.

[0005] The object of the present invention is to achieve the following technical solution: a load-adaptive distributed spatial text stream data processing system, the system comprising:

[0006] The data input layer is used to read the streaming spatial text data objects and spatial queries, and after parsing and preprocessing, obtain a spatial text data object four-element list containing an identifier, a geographic location, text data, and a generation time, and a spatial query four-element list containing an identifier, a query range, a keyword, and a query duration;

[0007] The partition routing layer performs routing operations of spatial partitions according to the streaming spatial text data objects and the quaternary list of spatial queries from the data input layer;

[0008] The matching processing layer stores and continuously matches the spatial text data objects that satisfy the query spatial range and the query keyword set based on the spatial partitioning of the partition routing layer during the duration of the spatial query;

[0009] The load balancing layer is used to monitor and collect the load information of the matching processing layer in real time, and when the load is unbalanced, it calls the load adaptation strategy and re-divides the local space partitions to meet the load balancing requirements;

[0010] The data storage layer is used to store outdated data in the matching processing layer.

[0011] Furthermore, the data input layer parses the stream data of the spatial text data object and processes it into a four-element list that is easy to operate later; the spatial text data object is text data containing position information, which is processed into a four-element list L o =[oid,loc,text,ts], oid is the data object identifier, loc is the geographic location, text is the text data, and ts is the generation time; the spatial query is a filtering query containing spatial range and keyword restrictions, which is processed into a four-element list L q =[qid, MBR, text, t], qid is the query identifier, MBR is the rectangular boundary of the query space range, text is the query keyword set, and t is the query duration.

[0012] Furthermore, the partition routing layer indexes and searches spatial partitions for spatial text data objects and spatial queries; each spatial partition corresponds to a rectangular area composed of continuous cells, and each cell in the spatial partition contains identifier information and boundary information of the spatial partition; the loc information of the spatial text data object determines a unique spatial partition, and the spatial range of the query covers multiple spatial partitions.

[0013] Furthermore, the division of the rectangular area occupied by the spatial partition has an order, firstly, the first dimension is divided horizontally or vertically to form candidate spatial partitions, and then the candidate spatial partitions are divided in another dimension to form spatial partitions:

[0014] Define two pairs (xmin, ymin) and (xmax, ymax) to represent the positions of spatial partitions. (xmin, ymin) is the coordinate of the bottom-left cell of the spatial partition, and (xmax, ymax) is the coordinate of the top-right cell of the spatial partition. When traversing the covered spatial partitions during a spatial query, move right by the method of <y remains unchanged, xmax + 1> to traverse other spatial partitions in the same candidate spatial partition, and then change x to the abscissa of the first main cell of this candidate spatial partition, and perform the operation of ymin - 1 on y to move down to reach other candidate spatial partitions, so as to traverse all the spatial partitions covered by the spatial query;

[0015] Define a main cell, which is the top-left cell of the spatial partition covered by the spatial query. Only access the main cell during the adjacent spatial partition search.

[0016] Further, during the adjacent search, when moving right to a spatial partition not covered by the query spatial range, change the y value to enter other candidate spatial partitions for traversal; when moving down to a spatial partition not covered by the query spatial range, the routing operation ends.

[0017] Further, the matching processing layer consists of multiple processing units. Each spatial partition corresponds to a processing unit, and each processing unit maintains a local keyword index; when the geographical location of the spatial text data object satisfies the spatial range of the spatial query and the text data information of the spatial text data object contains all the keywords of the spatial query, match this spatial text data object for this spatial query.

[0018] Further, the matching processing layer divides time slots for the streaming spatial text data objects, updates the time slots at regular intervals, and the older the data, the coarser the granularity; the granularity follows the exponential function f(x) = 2 x , where x is the survival time of the data in the matching processing layer; regularly check the data size in the processing unit, and if it exceeds the predefined threshold, store the oldest time slot data in the data storage layer.

[0019] Further, the load balancing layer monitors and collects the total load information W i and the dimensional load distribution table Wd i of each processing unit in the matching processing layer in real time. When the ratio of the maximum load to the minimum load in the processing unit W max / W min > δ, based on the load adaptive strategy, transfer the load in the processing unit P with the maximum load max to the adjacent processing unit of this processing unit; where the dimensional load distribution table Wd i statistics the load distribution of this processing unit in the second dimension.

[0020] Furthermore, when the load of the matching processing layer is unbalanced, the local space partitions near the maximum load processing unit are re-divided within the same candidate space partition, and the process is as follows:

[0021] (1) According to P max The two-tuple coordinates (xmin, ymin), (xmax, ymax) are determined by P max The spatial partitions where the adjacent processing units in the same candidate spatial partition are located;

[0022] (2) P max The local space redivision operation is performed with two adjacent processing units. The total load of the three adjacent space partitions is calculated first, and then the average load of each space partition is obtained. The maximum load of the three after redivision is taken as W. max , if W is no longer satisfied max / W min >δ, then proceed to step (3), otherwise expand the scope of local space redivision and add P max After the adjacent processing units of the two adjacent processing units are determined, it is determined whether the balancing trigger condition is met. If not, step (3) is performed. If it is met, the scope of the local space redivision is expanded again, and so on, until the redivisioned space partitions do not meet the balancing trigger condition.

[0023] (3) When redividing, merge the dimensional load distribution table Wd of n processing units from left to right in spatial order i1 ,Wd i2 ,…,Wd in , n processing units need to be split n-1 times and redivided into n spatial partitions; the specific splitting process is: accumulate the dimension loads from left to right on the merged dimension load distribution table. When the first accumulated sum is greater than or equal to the average load, the first split is performed, and the accumulation continues without clearing. When the second accumulated sum is greater than or equal to twice the average load, the second split is performed, and so on, until n-1 splits are performed.

[0024] Furthermore, the data storage layer releases obsolete data from the memory in the processing unit of the matching processing layer, writes it to a persistent storage, and connects it with other peripheral systems for future processing and analysis, wherein the peripheral systems include RDMS, HDFS, HBase, and Hive.

[0025] The advantages and beneficial effects of the present invention are:

[0026] 1. It can reduce the total load of all processing units and improve real-time performance and throughput.

[0027] The present invention uses an enhanced grid of the Neighbor-Based routing algorithm when routing spatial text data objects and queries in spatial partitions. The search complexity does not depend on the granularity of the grid, but only on the number of neighboring spatial partitions, which greatly reduces the routing overhead. The streaming data in the processing unit is divided into time slots, and outdated data is regularly migrated, which improves real-time performance and throughput.

[0028] 2. Dynamically adjust the load and greatly reduce the overhead of load migration.

[0029] The present invention adds a load balancing layer, which can monitor the load changes of each processing unit in real time. The load balancing mechanism is incremental, that is, it does not redefine all partitions, but simply shifts, splits and merges operations to update a few adjacent partitions, which greatly reduces the overhead of load migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The present invention is further described below in conjunction with the accompanying drawings and embodiments;

[0031] Figure 1 A load-adaptive distributed spatial text stream data processing system of the present invention;

[0032] Figure 2 A diagram illustrating the division and search of spatial partitions of the partitioned routing layer of the present invention;

[0033] Figure 3 This is a schematic diagram of load balancing space partitioning of the present invention. DETAILED DESCRIPTION

[0034] To make the objectives, technical solutions and advantages of the embodiments of the present invention more clear, the implementation modes of the present invention are described in detail below with reference to the accompanying drawings and embodiments.

[0035] like Figure 1 As shown in Figure 1, this system consists of five parts, namely data input layer, partition routing layer, matching processing layer, load balancing layer and data storage layer. The input data of the system is specific stream data, such as posts with location information on social platforms, search terms in search engines, and ride requests initiated by customers of taxi apps.

[0036] First, the spatial text data objects and spatial queries carrying geographic location information are connected to the data input layer for parsing and preprocessing.

[0037] Next, the processed stream data enters the partition routing layer, which routes the spatial text data objects and spatial queries to the processing units of the relevant spatial partitions.

[0038] Then, the spatial query and the spatial text data object are matched on the processing unit, and the streaming data is divided into time slots to transfer outdated data.

[0039] The data storage layer interfaces the outdated data from the matching processing layer processing units with other peripheral file systems for future analysis.

[0040] Finally, the load balancing layer monitors the load changes on the processing units in real time. When the load is unbalanced, it uses the space partition routing algorithm to search the neighborhood and calls the balancing strategy to transfer the load.

[0041] The detailed introduction of each subsystem in the present invention is as follows:

[0042] 1. Data input layer

[0043] It is mainly used to read streaming spatial text data objects and spatial queries and parse and preprocess the original data stream to obtain a four-element list containing identifiers, geographic locations, text data and generation time, as well as a spatial query four-element list containing identifiers, query ranges, keywords and query duration.

[0044] Specifically, the data input layer processes the given stream data as follows:

[0045] Extract the identifier, geographic location, text data and generation time information contained in the original stream data and represent them as a four-element list. The spatial text data object is text data containing location information, which is processed into a four-element list L o =[oid,loc,text,ts], oid is the data object identifier, loc is the geographic location, text is the text data, and ts is the generation time. The spatial query is a filtering query containing spatial range and keyword restrictions, which is processed into a four-element list L q =[qid, MBR, text, t], qid is the query identifier, MBR is the rectangular boundary of the query space range, text is the query keyword set, and t is the query duration.

[0046] 2. Partition routing layer

[0047] The routing operation of spatial partitioning is mainly based on the streaming spatial text data objects from the data input layer and the quad list of spatial query.

[0048] Specifically, Figure 2 As shown in the figure, the partition routing layer processes streaming spatial text data objects and spatial query allocation and search space partitioning as follows:

[0049] First, the partition routing layer divides the space into the first dimension in the y direction to form candidate space partitions RS1, RS2, RS3, ..., and then i Divide in the x direction to form the final spatial partition S ij, each spatial partition is a rectangular area containing a continuous number of cells, and the spatial partitions with the same i value belong to the same candidate spatial partition RS i . At this time, the partition routing layer divides the space into non-overlapping rectangular spatial partitions S ij , a rectangular spatial partition S ij corresponds to a processing unit P of the matching processing layer ij . Since the number of processing units P ij is huge in practice, the partition routing layer is used to index the processing units of the matching processing layer and allocate relevant processing units for the streaming spatial text data object and the spatial query. Through the non-overlapping spatial partitions S ij , the partition routing layer can directly distribute the spatial text data object to a single processing unit P in the spatial partition where it is located according to the loc information of the spatial text data object ij .

[0050] The spatial range MBR of the spatial query may overlap with multiple spatial partitions S ij , so it is necessary to route the spatial query to multiple processing units. For the routing operation of the spatial query, we use the spatial partition routing algorithm to establish the index and search of the processing unit P ij . Each processing unit P ij The spatial partition S where it is located ij can be represented as two pairs on the grid area. The first pair (xmin, ymin) represents the coordinates of the bottom left cell of the grid area, and the second pair (xmax, ymax) represents the coordinates of the top right cell of the grid area. The coordinates of two points can determine a grid area, that is, the corresponding spatial partition. Each cell in the grid area maintains the identifier information and boundary information of the corresponding spatial partition

[0051] The main cell is the top left cell of the spatial partition covered by the spatial query. When searching for adjacent spatial partitions, we only need to access the main cell in each spatial partition to know the identifier and boundary information of the spatial partition, and then move to the right by the method of <y remains unchanged, xmax + 1> to traverse other spatial partitions in the same candidate spatial partition, and then change x to the abscissa of the first main cell of this candidate spatial partition and perform the operation of ymin - 1 on y to move down to reach other candidate spatial partitions, so as to traverse all spatial partitions covered by the spatial query. When searching for adjacent spatial partitions, when accessing a spatial partition that is not covered by the query spatial range along the grid to the right, change the y value to enter other candidate spatial partitions for traversal; when accessing a cell that is not covered by the query spatial range downward, it marks the end of the routing of the spatial query

[0052] For example Figure 2When searching for spatial partitions covered by MBR in the spatial query, we start the search from the main cell (1,8) and get the spatial partition identifier S where the main cell is located. 11 And the boundary information of the spatial partition (0,5), (1,9), then search rightward in the same candidate spatial partition RS1, use the xmax+1 of the spatial partition boundary information as the new horizontal coordinate, and the vertical coordinate remains unchanged to reach the main cell (2,8) of the next spatial partition, and obtain the spatial partition identifier S where this main cell is located. 12 And the boundary information of the spatial partition (2,5), (6,9), take the xmax+1 of the spatial partition boundary information as the new horizontal coordinate, the vertical coordinate remains unchanged, and arrive at the main cell (7,8) of the next spatial partition, and obtain the spatial partition identifier S where this main cell is located 13 and the boundary information of the spatial partition (7,5), (9,9), and continue to visit the main cell (10,8) of the next spatial partition to the right. The spatial partition covered by the spatial query MBR is exceeded, and the search for the candidate spatial partition RS1 ends. Change x to the horizontal coordinate of the first main cell (1,8) of this candidate spatial partition, perform ymin-1 operation on y, move downward, and reach the first main cell (1,4) of the next candidate spatial partition RS2, and obtain the spatial partition identifier S where this main cell is located. 21 And the boundary information of the spatial partition (0,3), (4,4), and then search rightward and downward to obtain the covered spatial partition identifier S 22 ,S 31 ,S 32 ,S 33 ,When traversing further downward, the main cell (1,1) of the next candidate spatial partition is obtained, which is not within the spatial range covered by the spatial query MBR, and the routing operation ends.

[0053] 3. Matching processing layer

[0054] It is mainly used to store and continuously match spatial text data objects that meet the query spatial scope and query keyword set during the duration of the spatial query.

[0055] Specifically, the matching processing layer is composed of multiple processing units, each spatial partition corresponds to a processing unit, and each processing unit maintains a local keyword index. When the geographical location of the spatial text data object meets the spatial scope of the spatial query and the text information of the spatial text data object contains all the keywords of the spatial query, the spatial text data object is matched for the spatial query. The matching processing layer divides the incoming streaming spatial text data objects into time slots and updates the time slots at regular intervals. The older the data, the coarser the granularity. The time interval for updating the time slot can be 1 hour, 3 hours, 6 hours, etc.; the granularity follows the exponential function f(x)=2x , where x is the survival time of the data in the matching processing layer. The data size in the processing unit is checked regularly. If it exceeds a predefined threshold, such as 50% of the total storage, the oldest time slot is stored in the data storage layer.

[0056] 4. Load balancing layer

[0057] It is mainly used to monitor and collect the load information of each processing module in real time, and when the load is unbalanced, call the load adaptation strategy to re-divide the local space partitions within the same candidate space partition to meet the load balancing.

[0058] Specifically, given a time period, a processing unit W ij The load size is W ij =c1·|O ij |·|Q ij i |+c2·|O ij |+c3·|Q ij i |+c4·|Q ij d | wherein said O ij is the set of spatial text data objects in the processing unit, Q ij i is a set of query insertion requests for a given time period, Q ij d is the set of query deletion requests in a given time period, c1 is the matching cost of a query and a spatial text data object, c2 is the average cost of processing a spatial text data object, c3 is the average cost of processing query insertion, and c4 is the average cost of processing query deletion. The load balancing layer monitors and collects the total load information W of each processing module in real time. ij And dimension load distribution table Wd ij , when the ratio of the maximum load in the processing unit to the minimum load in the processing unit satisfies W max / W min >δ balance trigger condition, transfer the maximum load processing unit P max to the processing unit's neighboring processing units.

[0059] (1) According to P max The two-tuple coordinates (xmin, ymin), (xmax, ymax) can be determined with P max The spatial partitions where the adjacent processing units in the same candidate spatial partition are located;

[0060] (2) P maxThe local space redivision operation is performed with two adjacent processing units. The total load of the three space partitions is calculated first, and then the average load of each space partition is obtained. Step (3) is previewed, and the maximum load of the three after redivision is taken as W. max , if W is no longer satisfied max / W min >δ, then proceed to step (3), otherwise expand the scope of local space redivision and add P max After the adjacent two processing units of the adjacent two processing units are determined, it is determined whether the balance trigger condition is met. If not, step (3) is performed again. If it is met, the scope of the local space redivision is expanded again, and so on, until the redivisioned space partition does not meet the balance trigger condition;

[0061] (3) When redividing, merge the dimensional load distribution table Wd of n processing units from left to right in spatial order i1 ,Wd i2 ,…,Wd in , n processing units need to be split n-1 times and redivided into n spatial partitions. The specific splitting process is: accumulate the dimension loads from left to right in the merged dimension load distribution table. When the first accumulation is greater than or equal to the average load, the first split is performed, and the accumulation is continued without clearing. When the second accumulation is greater than or equal to twice the average load, the second split is performed, and so on, until n-1 splits are performed.

[0062] like Figure 3 As shown in the figure, the dimensional load distribution table of each processing module records the distribution of the load on the horizontal axis within the space partition. The space partition S ij It consists of three columns of cells. The sum of the loads of each column of cells is counted and monitored to obtain the dimensional load distribution table Wd of the processing module. ij =[8,2,10] and total load information W ij =20. When due to space partition S ij The processing unit P ij Overload operation becomes P max When the equilibrium trigger condition is met, ij and the adjacent processing units P located in the same candidate spatial partition i(j-1) , P i(j+1) Perform local space redivision operations. ij The spatial partition S ij The two tuples of coordinates (xmin, ymin), (xmax, ymax) are obtained by<xmin-1,ymax> and<xmax+1,ymax> Reach the left and right spatial partitions respectively and obtain their spatial partition identifiers S i(j-1) , S i(j+1)And dimension load distribution table Wd i(j-1) ([2,2,4,2,8]), Wd i(j+1) ([2,4,2,2]), the total load of the three is 40, and the average load is 13.3. Preview step (3), perform two splits, and obtain the total load information W of the processing module after load balancing i(j-1) , W ij and W i(j+1) 18, 14, and 8 respectively; if the maximum load Wmax = 18, the balancing trigger condition is no longer met, then execute step (3), otherwise, expand the scope of local space redivision and add S i(j-2) , S i(j+2) Calculate the total load and average load of the five spatial partitions and rehearse step (3), and so on, until W is no longer satisfied. max / W min >δ's equilibrium trigger condition.

[0063] 5. Data storage layer

[0064] It is mainly used to store outdated data in the processing modules of the matching processing layer.

[0065] Specifically, the data storage layer releases outdated data from the memory in the processing unit and writes it to persistent storage, which is connected to other peripheral systems (such as RDMS, HDFS, HBase, Hive, etc.) for future processing and analysis.

[0066] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A load-adaptive distributed spatial text stream data processing system, characterized in that: The system includes: The data input layer is used to read the streaming spatial text data objects and spatial queries, and after parsing and preprocessing, obtain a spatial text data object four-element list containing an identifier, a geographic location, text data, and a generation time, and a spatial query four-element list containing an identifier, a query range, a keyword, and a query duration; The partition routing layer performs routing operations on spatial partitions based on the streaming spatial text data objects and the quaternary list of spatial queries from the data input layer. The division of the rectangular area occupied by the spatial partition has a sequence. First, the first dimension x direction is segmented horizontally or vertically to form candidate spatial partitions, and then the candidate spatial partitions are segmented in another dimension y direction to form spatial partitions: Define two pairs (x min ,y min ), (x max ,y max ) to indicate the location of the spatial partition, (x min ,y min ) is the coordinate of the lower left corner cell of the spatial partition, (x max ,y max ) is the coordinate of the upper right corner cell of the spatial partition; when the spatial query traverses the covered spatial partition, it keeps y unchanged and performs x max The +1 operation method moves right, traverses the cells in the same candidate space partition, and then changes x to the horizontal coordinate of the first main cell of this candidate space partition, and performs y on y min -1 operation, moving downward to other candidate spatial partitions, thereby traversing all spatial partitions covered by the spatial query; Define a primary cell, which is the top left cell of the spatial partition covered by the spatial query; only visit the primary cell when searching for adjacent spatial partitions; The matching processing layer is based on the spatial partitions of the partition routing layer. Each spatial partition corresponds to a processing unit. During the duration of the spatial query, the spatial text data objects that meet the query spatial range and the query keyword set are stored and continuously matched; The load balancing layer is used to monitor and collect the load information of the matching processing layer in real time, including the total load information W of each processing unit. i And dimension load distribution table Wd i , and when the load is unbalanced, that is, when the ratio of the largest load in the processing unit to the smallest load in the processing unit is W max / W min > δ When the load adaptation strategy is called, the local space partition is redivided to transfer the maximum load processing unit P max The load in the processing unit is transferred to the adjacent processing unit of the processing unit, where the dimension load distribution table Wd i The load distribution of the processing unit in the second dimension is counted to meet the load balancing requirement. When the load of the matching processing layer is unbalanced, the local space partition near the maximum load processing unit is re-divided in the same candidate space partition. The process is as follows: (1) According to P max The two-tuple coordinates (x min ,y min ), (x max ,y max ) Determine the max The spatial partitions where the adjacent processing units in the same candidate spatial partition are located; (2) P max The local space redivision operation is performed with two adjacent processing units. The total load of the three adjacent space partitions is calculated first, and then the average load of each space partition is obtained. The maximum load of the three after redivision is taken as W. max , if W is no longer satisfied max / W min > δ If the equilibrium trigger condition is met, then proceed to step (3), otherwise expand the scope of local space redivision and add P max After the adjacent processing units of the two adjacent processing units are divided, it is determined whether the balance trigger condition is met. If not, step (3) is performed. If it is met, the scope of the local space redivision is expanded again, and so on, until the redivision space partition does not meet the balance trigger condition; (3) When redividing, merge the dimensional load distribution table Wd of n processing units from left to right in spatial order i1 ,Wd i2 ,…,Wd in , n processing units need to be split n-1 times and re-divided into n spatial partitions; the specific splitting process is: accumulate the dimension loads from left to right on the merged dimension load distribution table. When the first accumulated sum is greater than or equal to the average load, the first split is performed, and the accumulation continues without clearing. When the second accumulated sum is greater than or equal to twice the average load, the second split is performed, and so on, until n-1 splits are performed; The data storage layer is used to store outdated data in the matching processing layer.

2. The load-adaptive distributed spatial text stream data processing system according to claim 1, characterized in that: The data input layer parses the stream data of the spatial text data object and processes it into a four-element list that is easy to operate later; the spatial text data object is text data containing position information, which is processed into a four-element list L o =[oid,loc, text, ts], oid is the data object identifier, loc is the geographic location, text is the text data, and ts is the generation time; the spatial query is a filtering query containing spatial range and keyword restrictions, which is processed into a four-element list L q = [qid, MBR, text, t], qid is the query identifier, MBR is the rectangular boundary of the query space range, text is the query keyword set, and t is the query duration.

3. A load-adaptive distributed spatial text stream data processing system as claimed in claim 1, characterized in that: The partition routing layer indexes and searches spatial partitions for spatial text data objects and spatial queries; each spatial partition corresponds to a rectangular area composed of continuous cells, and each cell in the spatial partition contains the identifier information and boundary information of the spatial partition; the loc information of the spatial text data object determines the unique spatial partition, and the spatial range of the query will cover multiple spatial partitions.

4. A load-adaptive distributed spatial text stream data processing system as claimed in claim 1, characterized in that: During the neighboring search, when moving rightward to a spatial partition not covered by the query spatial range, the y value is changed to enter other candidate spatial partitions for traversal; when moving downward to a spatial partition not covered by the query spatial range, the routing operation ends.

5. A load-adaptive distributed spatial text stream data processing system as claimed in claim 1, characterized in that: The matching processing layer is composed of multiple processing units, each of which maintains a local keyword index; when the geographical location of the spatial text data object meets the spatial scope of the spatial query and the text data information of the spatial text data object contains all the keywords of the spatial query, the spatial text data object is matched for the spatial query.

6. The load-adaptive distributed spatial text stream data processing system according to claim 5, characterized in that: The matching processing layer divides the streaming spatial text data objects into time slots, and updates the time slots at regular intervals. The older the data, the coarser the granularity. The granularity follows the exponential function f(x) = 2 x , where x is the survival time of the data in the matching processing layer; the data size in the processing unit is checked regularly, and if it exceeds the predefined threshold, the oldest time slot data is stored in the data storage layer.

7. The load-adaptive distributed spatial text stream data processing system according to claim 1, characterized in that: The data storage layer releases obsolete data from the memory in the processing unit of the matching processing layer, writes it into persistent storage, and connects it with other peripheral systems for future processing and analysis, and the peripheral systems include RDMS, HDFS, HBase, and Hive.

Images