A full-net companion method based on space-time grid data compression

Abstract

The application provides a kind of whole network accompanying method based on space-time grid data compression. Based on space-time position data resources, using big data technology, space-time grid data compression technology and the like, space-time trajectory data is analyzed and processed, including position data extraction, original trajectory vectorization, trajectory vector filling, space-time bucketing and trajectory merging and the like, and the accompanying behavior relationship existing in space-time position data is calculated at one time. The application can solve the problems of long computation time of existing methods, inability to realize long-term accompanying analysis, difficulty in realizing the value of space-time data and the like, and has the advantages of low computational complexity, long-term accompanying and maximum realization of the value of space-time data.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of big data analysis, and particularly relates to a whole-network accompanying method based on space-time grid data compression. BACKGROUND

[0002] With the continuous popularization of 5G and the continuous popularity of sensor devices, more and more location information data is collected, and the location data set is becoming larger and larger. It has always been a hot issue to classify the huge location data in a targeted manner within a limited time, and it is the main problem of accompanying relationship discovery and close contact tracking.

[0003] The existing space-time accompanying algorithm mainly adopts the idea of machine learning to calculate the accompanying of a target and a short period. The scene targeted is the unique identification of the known target in the space-time data, and the accompanying relationship with the user in the specified time period is calculated. The main problems of the existing algorithm are: 1. The online calculation method is adopted, and if many user subjects need to be calculated at the same time, the calculation time will be long, the calculation resource occupation will be large, and other tasks may not be run during this period; 2. Since the short-time and target accompanying calculation method is adopted, all the accompanying relationships in the space-time data cannot be accumulated, and more valuable information cannot be further mined. SUMMARY

[0004] In order to overcome the shortcomings of the prior art, the application provides a whole-network accompanying method based on space-time grid data compression. Based on space-time location data resources, the space-time trajectory data is analyzed and processed by using big data technology, space-time grid data compression technology and the like, including position data extraction, original trajectory vectorization, trajectory vector filling, space-time bucketing and trajectory merging processes, and the accompanying behavior relationship existing in the space-time location data is calculated at one time. The application can solve the problems of long accompanying behavior calculation time, inability to realize long-term accompanying analysis and difficulty in playing the value of space-time data of the existing method, and has the advantages of low calculation complexity, long-term accompanying and maximum play of the value of space-time data.

[0005] A whole-network accompanying method based on space-time grid data compression, characterized in that the steps are as follows:

[0006] Step 1, position data extraction: the position data of each user subject per day is extracted from the position data source, and the position data is grouped according to the unique identification id of the user subject, time, and the encoding of the 13-level Google S2 block corresponding to the latitude and longitude value of each data, wherein the time is accurate to the minute; the number of times each S2 block appears in the same minute is counted, and then the S2 block with the most occurrences is taken as the position block in the minute, and for each user subject, only one position data is obtained per minute, and the data extraction is completed;

[0007] Step 2, original trajectory vectorization: fill the extracted position data of each user subject into the trajectory vector in time sequence, and the position data of each minute corresponds to a node in the trajectory vector. If the position data of the minute is missing, the node is 0. The index of the node in the vector is the minute index, i.e. 1, 2, …, 1440. Thus, the initial trajectory vector is formed, and the length of the initial trajectory vector is 1440.

[0008] Step 3, trajectory vector filling: fill the position nodes in the initial trajectory vector with continuous missing time less than 15 minutes according to the nearest principle. Specifically, let p k , p k+n be two adjacent nodes in the initial trajectory vector, k represent the index of the node in the trajectory vector, and n be the index difference between the two adjacent nodes, n≤15. The i-th node p i after filling is obtained according to the following formula:

[0009]

[0010] Step 4, space-time bucketing: for the filled trajectory vector, take a position node every five minutes, and according to the different starting positions of the taken nodes, obtain five different sub-trajectory vectors with a length of 288; for each sub-trajectory vector, obtain the position nodes by sliding the time window of 30 minutes; each sub-trajectory vector obtains 283 trajectory vectors with a time length of 30 minutes and 5 trajectory vectors with a time length less than 30 minutes; delete the trajectory vectors containing nodes of 0, and process the remaining sliding window trajectory vectors according to the following process:

[0011] Step a: for the sliding window trajectory vector with a time length of 30 minutes, calculate its space-time bucket id according to the following formula:

[0012] id=index_p / 5 (2)

[0013] Wherein, index_p represents the node index of the first node p in the initial trajectory vector in the sliding window trajectory vector;

[0014] Then, the time-space bucket recording is performed, and the time-space bucket RDD data structure is [(time-space bucket id, (trajectory vector, user subject id))].

[0015] Step b: for all sliding window trajectory vectors with a moving distance exceeding 5 kilometers, the shortest trajectory vector with a moving distance exceeding 5 kilometers is intercepted from the first node, the time-space bucket id is calculated according to the above formula, and the time-space bucket recording is performed;

[0016] Step c: for each time-space bucket, the RDD data structure is transformed into [(time-space bucket id, trajectory vector), user subject id], then the aggregation operator is used to aggregate all time-space bucket RDD data with the same key, and the accompanying relationship user array is obtained, the RDD data structure of which is [((time-space bucket id, accompanying trajectory vector), [accompanying relationship user array]), then the user subject ids in the accompanying relationship user array are combined two by two, and the combined RDD data structure is flattened to obtain a plurality of accompanying relationship pair arrays, the RDD data structure of which is [((time-space bucket id, accompanying trajectory vector), user subject id1-user subject id2)];

[0017] Step 5, trajectory merging: the RDD data of the accompanying relationship pair array is aggregated using the aggregation operator, the value array of the aggregated RDD data is arranged in ascending order, so that the array is arranged in time sequence, and it is judged whether the adjacent time-space bucket accompanying paths after arrangement are continuous paths, if yes, they are combined into a path in time sequence, and the accompanying time of the combined path is taken as the starting time, otherwise, the original path starting time is directly taken as the accompanying time, after the processing is completed, the RDD data structure of the newly obtained accompanying relationship pair array is flattened to obtain the final accompanying relationship pair result.

[0018] The beneficial effects of the present application are: by performing position data extraction, the path burr can be removed; by performing trajectory vectorization and filling processing, a unified standard is provided for mass time-space data calculation, and on the basis of not changing the authenticity of the trajectory data, the trajectory can be described in the greatest detail, and the coverage rate of the accompanying calculation is improved; since the time bucket method is adopted, all user trajectories are sliced and recorded in the specified time-space bucket according to the time node, which reduces the calculation complexity and makes the network-wide user trajectory accompanying calculation possible. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a network-wide accompanying method flowchart based on time-space grid data compression of the present application;

[0020] Figure 2 is a trajectory vectorization process schematic diagram;

[0021] Figure 3 is a schematic diagram of a trajectory vector filling process;

[0022] Figure 4 is a schematic diagram of a 5-kilometer path splitting process;

[0023] Figure 5 is a schematic diagram of time bucketing;

[0024] Figure 6 is a schematic diagram of intra-bucket trajectory merging;

[0025] Figure 7 is a schematic diagram of inter-bucket trajectory merging. DETAILED DESCRIPTION

[0026] The application is further illustrated below in conjunction with the accompanying drawings and examples, and the application includes but is not limited to the following examples.

[0027] As shown in Figure 1 , the application provides a full-network accompanying method based on space-time grid data compression, and the specific implementation process is as follows:

[0028] 1. Position data extraction

[0029] The position data of each user subject per day is extracted from the position data source, and the position data is grouped according to the user subject unique identifier id, time, and the encoding of the 13-level Google S2 block corresponding to each data latitude and longitude value by using a big data component, wherein the time is accurate to the minute; the number of occurrences of each S2 block in the same minute is counted, and then the S2 block with the most occurrences is taken as the position block in the minute. For each user subject, only one piece of position data corresponds to each minute, the data extraction is completed, and the deburring processing is realized.

[0030] 2. Original trajectory vectorization

[0031] The position data of each user subject after extraction is filled into the trajectory vector in chronological order, and the position data of each minute corresponds to a node in the trajectory vector. If the position data of the minute is missing, the node is 0. The index of the node in the vector is the minute index, i.e. 1, 2, …, 1440. In this way, an initial trajectory vector is formed, and the length of the initial trajectory vector is 1440, corresponding to 1440 minutes of a day. The specific process is shown in Figure 2 .

[0032] After this process, the 24-hour day is described as a 1440-length vector from the time dimension, so that the trajectory of each user in the full network can be described as this trajectory vector, and a unified processing standard is provided for massive space-time data calculation.

[0033] 3. Trajectory vector filling

[0034] Since the spatiotemporal trajectory vector often does not have position information every minute, the present application proposes a trajectory filling method, within a defined time span threshold, to complete the information-empty position node by the nearest principle, to describe the trajectory in the greatest detail on the basis of not changing the authenticity of the trajectory data, and to improve the coverage rate of the accompanying calculation. Specifically,

[0035] The initial trajectory vector formed by the original trajectory vectorization is filled, and the position node with a continuous loss time of more than 15 minutes is not processed during filling. The position node with a continuous loss time of less than 15 minutes in the initial trajectory vector is filled according to the nearest principle: let p k and p k+n be two adjacent nodes in the initial trajectory vector, k represents the node index in the trajectory vector, and n is the index difference between the two adjacent non-zero nodes, n≤15. The i-th node p i after filling is obtained according to the following formula:

[0036]

[0037] Figure 3 A filling example diagram is given.

[0038] 4. Spatiotemporal bucketing

[0039] For the filled trajectory vector, a position node is taken every five minutes, and according to the difference of the starting position of the taken node, five different sub-trajectory vectors with a length of 288 are obtained. These sub-trajectories can fully cover the 1440-length trajectory, and no position node will be missed during operation, which can improve the coverage rate of the accompanying behavior calculation.

[0040] For each sub-trajectory vector, a position node is obtained by sliding according to a time window of 30 minutes. Each sub-trajectory vector obtains 283 trajectory vectors with a time length of 30 minutes and five trajectory vectors with a time length of less than 30 minutes. As Figure 4 shown in the figure, spatiotemporal bucket 0 represents the 0th coded spatiotemporal bucket, and the trajectory “x-a-b-f-g-l,..” behind represents the trajectory category in the 0th bucket. The essence of this is to slice the trajectory from the time dimension to reduce the complexity of subsequent trajectory comparison.

[0041] For all the obtained sliding window trajectory vectors, delete the trajectory vectors containing 0 nodes, and perform spatiotemporal bucketing on the remaining sliding window trajectory vectors. The specific process is as follows:

[0042] Step a: for the sliding window trajectory vector with a time length of 30 minutes, calculate its spatiotemporal bucket id according to the following formula respectively:

[0043] id = index p / 5 (4)

[0044] wherein, index p represents the node index of the first node p in the sliding window trajectory vector in the initial trajectory vector;

[0045] Then, the time-space bucketing is performed, and the RDD data structure of the time-space bucketing is [(time-space bucket id, (trajectory vector, user subject id))].

[0046] Step b: for all sliding window trajectory vectors with a moving distance exceeding 5 kilometers, the shortest trajectory vector with a moving distance exceeding 5 kilometers is cut from the first node thereof, the time-space bucket id thereof is calculated according to the foregoing formula (the time-space bucket id thereof is the same as that of the original trajectory), and the time-space bucketing is performed to obtain the time-space bucketing RDD data structure as shown above.

[0047] Specifically, assuming that there is a sliding window trajectory vector “x-c-v-b-n-m”, and the distance of “x-c-v-b” has exceeded 5 kilometers, therefore, “x-c-v-b” needs to be cut from the original trajectory, and it is worth noting that the shortest trajectory meeting 5 kilometers must be cut from the first node of the original trajectory, that is, the original trajectory and the shortest trajectory meeting 5 kilometers have the same time-space bucket id value, belong to the same time-space bucket, and have the same RDD data structure as the 30-minute trajectory. Figure 5 A 5-kilometer segment trajectory splitting diagram is shown.

[0048] After the above process, the bucketing operation of the filled trajectory vector is completed. The time-space bucketing operation is from the time slicing dimension, and the unique time slice corresponds to the unique time-space bucket. In the bucketing, all paths and users in the network time-space data in the time period are put in, which provides data support for one-time calculation of all accompanying relationships in the time-space data.

[0049] Step c: for each time-space bucket, the RDD data structure thereof is transformed into [(time-space bucket id, trajectory vector), user subject id], and then an aggregation operator is used to aggregate all time-space bucket RDD data with the same key (i.e., (time-space bucket id, trajectory vector)) to obtain an accompanying relationship user array, and the RDD data structure thereof is [((time-space bucket id, accompanying trajectory vector), [accompanying relationship user array])], as shown in Figure 6As shown, "T', T1, T2" represent the accompanying array after polymerization. Then, by combining the user subject id in the accompanying relationship user array two by two, a new RDD data structure is formed [((spatiotemporal bucket id, accompanying trajectory vector), [user subject id1-user subject id2, user subject id1-user subject id3,...])], and the combined RDD data structure is flattened to obtain a number of accompanying relationship pair arrays, and the RDD data structure is [((spatiotemporal bucket id, accompanying trajectory vector), user subject id1-user subject id2)].

[0050] 5. Trajectory merging

[0051] In order to accurately express the accompanying time and accompanying trajectory of the real accompanying behavior, it is necessary to merge the inter-bucket continuous paths of the same accompanying relationship pair to form long paths arranged in time sequence.

[0052] Therefore, first, the RDD data of the pair array is aggregated using the aggregation operator, the value array of the aggregated RDD data is arranged in ascending order, so that the array is arranged in chronological order, and it is judged whether the adjacent space-time bucket accompanying path after arrangement is a continuous path. If so, it is combined into a path in chronological order, and the starting time of the combined path is taken as the accompanying time, such as the RDD data structure after ascending arrangement [([0_x-a-b-f-g-l, 1_a-b-f-g-l-f,...], (T1-T2)), ([0_a-b-d-r-k-m, 1_b-d-r-k-m-f,...], (T4-T5))] (for convenience of expression and processing, the space-time bucket id and trajectory vector are connected by an underscore here, and T1, T2,... represent two user subject ids respectively). It can be seen that the trajectory of the 0th bucket and the trajectory of the 1st bucket have a continuous phenomenon, which is combined into a long accompanying trajectory "x-a-b-f-g-l-f". The starting point of this trajectory is the starting position "x" of the 0th bucket, and the ending position is the ending position "f" of the 1st bucket. The time corresponding to the position "a" is taken as the accompanying start time StartTime, and the time corresponding to the position "f" is taken as the accompanying end time EndTime, and T4-T5 is similar. If there is no continuous phenomenon, the original trajectory start time is directly taken as the accompanying time. After such processing, the obtained pair array RDD data structure of the accompanying relationship is, for example, the RDD data structure of the previous example is [([StartTime1->EndTime1->"x-a-b-f-g-l-f", StartTime2->EndTime2->"..."], (T1-T2))] and the like. The pair array RDD data structure of the new accompanying relationship is flattened to obtain the final several accompanying relationship pair results. For the previous example, the RDD data structure after flattening operation can be expressed as the following structure: [(StartTime1->EndTime1->"x-a-b-f-g-l-f", (T1-T2))], [(StartTime2->EndTime2->"... ", (T'-T))]. Figure 7 The accompanying relationship that needs to be merged between buckets is given, where T', T1, T2 represent the user subject id under the space-time bucket trajectory. After the flattening operation is completed, StartTime1->EndTime1 and StartTime2->EndTime2 are two time periods that do not exist. The intersection expresses the actual accompanying behavior.

[0053] All accompanying relationship pair results are persisted to the database, which can accumulate accompanying relationships, ensure query efficiency and concurrency through interface query, and can deeply analyze the behavior of all network users through analysis of long-term accompanying results.

Claims

1. A full-net companion method based on spatio-temporal mesh data compression, characterized by The steps are as follows: Step 1, position data extraction: the position data of each user subject per day is extracted from the position data source, and the position data is grouped according to the user subject unique identifier id, time, and the encoding of each data latitude and longitude value corresponding to the 13-level Google S2 block, wherein the time is accurate to the minute; the number of times each S2 block appears in the same minute is counted, and then the S2 block with the most occurrences is taken as the position block in the minute, and for each user subject, only one position data corresponds to each minute, completing data extraction; Step 2, original trajectory vectorization: fill the extracted position data of each user subject into the trajectory vector in time sequence, and the position data of each minute corresponds to a node in the trajectory vector, if the position data of the minute is missing, the node is 0, the index of the node in the vector is the minute index, that is, 1, 2, …, 1440, thus forming an initial trajectory vector, and the length of the initial trajectory vector is 1440; Step 3, trajectory vector filling: fill the position nodes with less than 15 minutes of continuous missing time in the initial trajectory vector according to the nearest principle, specifically: let p k , p k+n be two adjacent nodes in the initial trajectory vector that are not 0, k represents the node index in the trajectory vector, and n is the index difference between the two adjacent nodes that are not 0, n≤15, the i-th node p i after filling is obtained according to the following formula: Step 4, time-space bucketing: for the filled trajectory vector, take a position node every five minutes, and according to the different starting positions of the taken nodes, obtain 5 different sub-trajectory vectors with a length of 288; for each sub-trajectory vector, obtain 283 trajectory vectors with a time length of 30 minutes and 5 trajectory vectors with a time length less than 30 minutes by sliding the position nodes according to a time window of 30 minutes; delete the trajectory vectors containing nodes with 0, and process the remaining sliding window trajectory vectors as follows: Step a: for the sliding window trajectory vector with a time length of 30 minutes, calculate its time-space bucket id according to the following formula respectively: id=index_p / 5 (2) Wherein, index_p represents the node index of the first node p in the sliding window trajectory vector in the initial trajectory vector; Then perform time-space bucketing recording, and the RDD data structure of the time-space bucket is [(time-space bucket id, (trajectory vector, user subject id))]; Step b: for all sliding window trajectory vectors with a moving distance exceeding 5 kilometers, extract the shortest trajectory vector with a moving distance exceeding 5 kilometers from the first node, calculate its time-space bucket id according to the foregoing formula, and perform time-space bucketing recording; Step c: for each time-space bucket, transform its RDD data structure to [(time-space bucket id, trajectory vector), user subject id], and then use the aggregation operator to aggregate all time-space buckets with the same key RDD data to obtain the accompanying relationship user array, and the RDD data structure of the accompanying relationship user array is [((time-space bucket id, accompanying trajectory vector), [accompanying relationship user array])]; then, combine the user subject ids in the accompanying relationship user array two by two, and perform flat processing on the combined RDD data structure to obtain a number of accompanying relationship pair arrays, and the RDD data structure of the accompanying relationship pair array is [((time-space bucket id, accompanying trajectory vector), user subject id1-user subject id2)]; Step 5, track merging: using the aggregation operator to aggregate the RDD data of the array of the accompanying relationship pairs, arranging the value array of the aggregated RDD data in ascending order so that the array is arranged in chronological order, judging whether the adjacent spatiotemporal bucket accompanying paths after arrangement are continuous paths, if yes, combining them into a path in chronological order and taking the starting time of the combined path as the accompanying time, otherwise, directly taking the starting time of the original path as the accompanying time, and thus processing is completed, and then performing the flattening operation on the new obtained accompanying relationship pair array RDD data structure to obtain the final several accompanying relationship pair results.

Images