Warehousing processing method for GPS data

Abstract

The invention discloses a GPS data storage processing method, which belongs to the technical field of GPS data processing, and comprises the following steps: obtaining real historical GPS data information and road network data information of a user in a first preset time period, preprocessing the historical GPS data information and the road network data information to obtain a first data feature set and a second data feature set; performing feature fusion on the first data feature set and the second data feature set to construct a third data feature set; and constructing a GPS data authenticity identification model, sending the third data feature set into the GPS data authenticity identification model for training, obtaining the trained GPS data authenticity identification model, identifying the GPS data in a second preset time period, if the identification result is real GPS data, storing the real GPS data in a database, and if the identification result is forged GPS data, storing the forged GPS data in the database. If so, marking; according to the method, authenticity identification can be effectively carried out on the to-be-stored GPS data, and the data reliability is ensured.

Description

Technical Field

[0001] The present invention relates to the field of GPS data processing technology, and in particular to a GPS data storage processing method. Background Art

[0002] GPS (Global Positioning System) data is the geographic location and related information obtained by GPS receivers receiving and processing signals from multiple GPS satellites. GPS data storage refers to the process of storing the location, time, speed, direction, altitude and other information collected by GPS into a database system after processing, conversion and verification. Storage of GPS data helps to achieve centralized storage, management and analysis of data.

[0003] If forged GPS data is stored in the database, it may be used for illegal activities, such as financial fraud, privacy infringement, etc., which will not only cause harm to individual users, but may also endanger social stability. Therefore, it is very important to identify the authenticity of GPS data during the storage process. Summary of the invention

[0004] In view of the above-mentioned prior art, the present invention provides a method for storing and processing GPS data, which mainly solves the technical problems existing in the above-mentioned background technology.

[0005] To achieve the above object, the technical solution of the embodiment of the present invention is implemented as follows:

[0006] A method for storing and processing GPS data, comprising the following steps:

[0007] Acquire real historical GPS data information and road network data information of the user within a first preset time period, and pre-process the historical GPS data information and the road network data information to obtain a first data feature set and a second data feature set;

[0008] Fusing the first data feature set and the second data feature set to construct a third data feature set;

[0009] Constructing a GPS data authenticity identification model, sending the third data feature set into the GPS data authenticity identification model for training, and obtaining a trained GPS data authenticity identification model;

[0010] The GPS data of the second preset time period is identified by the trained GPS data authenticity identification model. If the identification result is authentic GPS data, the authentic GPS data is stored in a database. If the identification result is forged GPS data, it is marked.

[0011] Optionally, the first data feature set includes a time series feature vector, a position feature vector, and a speed feature vector.

[0012] Optionally, the acquiring of the road network data information of the user within the first preset time period and preprocessing the road network data information to obtain the second data feature set specifically includes:

[0013] Acquire the shape of the road the user drives on during the first preset time, and construct a road shape feature matrix based on the road shape, where the road shape includes a straight section, a curved section, an uphill section, and a downhill section;

[0014] Acquire a road speed limit in a driving area of ​​the user within a first preset time, and construct a road speed limit feature vector based on the road speed limit;

[0015] Acquiring traffic flow information of a driving area of ​​the user within a first preset time, and constructing a traffic flow feature vector based on the traffic flow information;

[0016] The second data feature set is obtained by pre-processing the road shape feature matrix, the road speed limit feature vector, and the traffic flow feature vector and fusing the features.

[0017] Optionally, the fusing the first data feature set and the second data feature set to construct a third data feature set specifically includes:

[0018] Normalizing the first data feature set and the second data feature set;

[0019] Dynamic time warping is used to align the normalized first data feature set and the second data feature set, and the aligned first data feature set and the second data feature set are fused using a weighted moving average method to obtain a third data feature set.

[0020] Optionally, the specific steps of aligning the first data feature set and the second data feature set after normalization by using dynamic time warping are:

[0021] Calculate the distance matrix between each point in the first data feature set and the second data feature set, and find an optimal path from the upper left corner to the lower right corner of the matrix;

[0022] The first data feature set and the second data feature set are aligned according to the optimal path.

[0023] Optionally, if the recognition result is real GPS data, storing the real GPS data in a database specifically includes:

[0024] The real GPS data is written into a cache queue, and when the cache queue is full, the real GPS data in the cache queue is written into a disk file, and the real GPS data in the disk file is imported into a database in batches at regular intervals.

[0025] Optionally, the process further includes converting the format of the real GPS data from binary to JSON format.

[0026] The beneficial effects of the present invention are:

[0027] Compared with ordinary GPS platforms, the present invention adds an anti-counterfeiting identification processing link to the GPS trajectory data while storing the GPS data, analyzes the accessed GPS data through multiple dimensions, prevents the front-end device from sending forged GPS data, and ensures the reliability of GPS data;

[0028] Compared with the traditional method of directly writing GPS data into a relational database (such as MySQL, Oracle), the three-level data storage structure of "memory-disk-database" and the data writing method of "read-write thread separation" proposed in the present invention have fast data writing speed, are not prone to data loss, and can restart the program and try again when an exception occurs. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of a GPS data storage processing method in the present invention;

[0030] Figure 2 Schematic diagram of the traditional process for writing GPS data into a database;

[0031] Figure 3 The figure is a flow chart of a GPS data storage processing method in the present invention. DETAILED DESCRIPTION

[0032] The technical solution of the present invention is further elaborated in detail below in conjunction with the drawings and specific embodiments of the specification. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. In the following description, the expression "some embodiments" is related to a subset of all possible embodiments, but it should be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0033] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features well known in the art are not described.

[0034] It should be understood that the present invention can be implemented in different forms and should not be interpreted as being limited to the embodiments proposed herein. On the contrary, providing these embodiments will make the disclosure thorough and complete, and the scope of the present invention will be fully conveyed to those skilled in the art. And the purpose of the terms used herein is only to describe specific embodiments and is not intended to be a limitation of the present invention. When used herein, the singular forms of "one", "one" and "said / the" are also intended to include plural forms, unless the context clearly indicates another way. It should also be understood that the terms "compose" and / or "include" when used in this specification determine the presence of the features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or groups. When used herein, the term "and / or" includes any and all combinations of the relevant listed items.

[0035] It should also be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "inside", "outside", "left", "right" and similar expressions used herein are for illustrative purposes only and are not intended to be the only implementation method.

[0036] In order to fully understand the present invention, a detailed structure will be proposed in the following description to illustrate the technical solution proposed by the present invention. The optional embodiments of the present invention are described in detail as follows, but in addition to these detailed descriptions, the present invention may also have other implementations.

[0037] Example

[0038] Please refer to the attached Figure 1 The present application provides a method for storing and processing GPS data, comprising the following steps:

[0039] Acquire real historical GPS data information and road network data information of the user within a first preset time period, and pre-process the historical GPS data information and the road network data information to obtain a first data feature set and a second data feature set;

[0040] The first data feature set includes a time series feature vector, a position feature vector, and a speed feature vector;

[0041] Specifically, the real historical GPS data information can come from the user's mobile device, the vehicle's GPS tracking system or other GPS recording devices; after obtaining the real historical GPS data information, the data is preprocessed, cleaned and converted into a unified format, wherein the time series feature vector includes the time information of each GPS data point, such as: timestamp, date, etc. These features can help the model identify the timing pattern and abnormal behavior of the data, such as abnormal time jumps or lack of regularity; the position feature vector includes the longitude and latitude information of each GPS data point. By analyzing the position data, the spatial distribution pattern and abnormal movement of the data can be identified, such as impossible short-term displacement or movement path that violates geographical restrictions; the speed feature vector includes the speed of each GPS data point, which helps to identify speed changes that do not conform to physical laws, such as instantaneous extreme speed changes, which may be a sign of forged data, that is: real GPS data usually shows a reasonable time series pattern, position changes that conform to geographical restrictions and speed changes that conform to physical laws, while forged data often show inconsistency or abnormality in these aspects. The time series feature vector, position feature vector, and speed feature vector are then fused through preprocessing to obtain the first data feature set. The preprocessing can use a feature selection algorithm (such as LASSO, random forest, etc.) to select the most representative features, and then feature fusion is performed through feature splicing. The fused features help the model to subsequently analyze and identify true and false GPS data from different dimensions.

[0042] The step of obtaining the road network data information of the user within the first preset time period and preprocessing the road network data information to obtain the second data feature set specifically includes:

[0043] Acquire the shape of the road the user drives on during the first preset time, and construct a road shape feature matrix based on the road shape, where the road shape includes a straight section, a curved section, an uphill section, and a downhill section;

[0044] Acquire a road speed limit in a driving area of ​​the user within a first preset time, and construct a road speed limit feature vector based on the road speed limit;

[0045] Acquiring traffic flow information of a driving area of ​​the user within a first preset time, and constructing a traffic flow feature vector based on the traffic flow information;

[0046] The second data feature set is obtained by pre-processing the road shape feature matrix, the road speed limit feature vector, and the traffic flow feature vector and fusing the features;

[0047] Specifically, the second data feature set is obtained from the road network data information. By analyzing the road shape feature matrix, it helps the model to identify whether the location information in the GPS data is consistent with the actual road shape, thereby judging the authenticity of the data. For example, if the GPS data shows that the vehicle has made a curved motion on a straight road, this may be a sign of forged data; by analyzing the road speed limit feature vector and the speed limit of the actual road, it can be determined whether the vehicle is speeding, thereby evaluating the rationality of the GPS data; by analyzing the traffic flow feature vector, it can be used to verify whether the time series characteristics in the GPS data match the actual traffic flow. For example, during peak hours, the traffic flow is large, and forged GPS data may show that during peak hours, vehicles can travel at non-peak speeds, which is inconsistent with the actual situation; then the road shape feature matrix, the road speed limit feature vector, and the traffic flow feature vector are pre-processed to perform feature fusion to obtain the second data feature set. The pre-processing can use feature selection algorithms (such as LASSO, random forest, etc.) to select the most representative features, and then perform feature fusion by feature splicing. The fused features help the model to analyze and identify true and false GPS data from different dimensions in the future.

[0048] Fusing the first data feature set and the second data feature set to construct a third data feature set;

[0049] Specifically, the first data feature set provides the basic spatiotemporal information of GPS data, which is the basis for judging the authenticity of GPS data; the second data feature set provides environmental information related to the road, and provides contextual environmental information for judging the authenticity of GPS data. By merging the first data feature set and the second data feature set, a more comprehensive GPS data feature set can be obtained. This feature set not only contains the basic spatiotemporal information of GPS data, but also contains environmental information related to the road; the fusion of the first data feature set and the second data feature set can improve the accuracy and robustness of the GPS data authenticity identification model, so that the model can analyze and judge the data from multiple angles. For example, by combining the road shape and GPS location information, forged GPS tracks can be more accurately identified; by combining traffic flow and speed information, abnormal speed changes during high-traffic periods can be identified.

[0050] Constructing a GPS data authenticity identification model, sending the third data feature set into the GPS data authenticity identification model for training, and obtaining a trained GPS data authenticity identification model;

[0051] Specifically, the third data feature set is sent to the GPS data authenticity recognition model for training. The model can be a support vector machine, a random forest, a neural network, etc., and then the parameters of the model are adjusted through an optimization algorithm to minimize the prediction error. During the training process, the performance of the model is evaluated through cross-validation and other methods to ensure that the recognition accuracy of the model is effectively improved, and finally a trained GPS data authenticity recognition model is obtained.

[0052] Identify the GPS data of the second preset time period by using the trained GPS data authenticity identification model, and if the identification result is authentic GPS data, store the authentic GPS data in a database; if the identification result is forged GPS data, mark it;

[0053] Specifically, the GPS data of the second preset time period is the GPS data to be stored after authenticity identification. After the authenticity identification of the GPS data of the second preset time period, if the identification result is real GPS data, it will be stored; if it is identified as forged data, it will be marked and wait for processing by the management personnel.

[0054] As an optional implementation, the step of fusing the first data feature set and the second data feature set to construct a third data feature set specifically includes:

[0055] Normalizing the first data feature set and the second data feature set;

[0056] Using dynamic time warping (DTW) to align the first data feature set and the second data feature set that have been normalized, and fusing the aligned first data feature set and the second data feature set using a weighted moving average method to obtain a third data feature set;

[0057] The specific steps of aligning the first data feature set and the second data feature set after normalization by using dynamic time warping are:

[0058] Calculate the distance matrix between each point in the first data feature set and the second data feature set, and find an optimal path from the upper left corner to the lower right corner of the matrix;

[0059] Aligning the first data feature set and the second data feature set according to the optimal path;

[0060] Exemplarily, the first data feature set includes a time series feature vector, a position feature vector, and a speed feature vector, which is recorded as a sequence A = {a1, a2, ..., a m}; The second data feature set includes a road shape feature matrix, a road speed limit feature vector, and a traffic flow feature vector, which is recorded as a sequence B = {b1, b2, ..., b n};

[0061] Use Euclidean distance to calculate the distance between each pair of points in the two sequences and construct an m×n distance matrix D, where D(i,j) represents the distance between the i-th point in sequence A and the j-th point in sequence B;

[0062] Starting from D(a1,b1), the shortest path to each (i,j) node is calculated by dynamic programming. During the calculation process, only the three points in the upper left corner (i.e., D(i-1,j), D(i-1,j-1), and D(i,j-1)) are considered, and the one with the shortest path is selected as the next node;

[0063] From D(a m ,b n ) backtrack to find the path from D(a1,b1) to D(a m ,b n ), which represents the optimal time axis alignment between the two sequences; calculate the path length of the optimal path, which is the DTW distance between the two sequences. The shorter the path length, the more similar the two sequences are; align the two feature sets according to the optimal path so that they are consistent in time series, laying the foundation for subsequent feature fusion;

[0064] It should be noted that although the first feature set and the second feature set are both selected from data of the same time period and have the same timestamp, if the data sampling frequencies of the first data feature set and the second data feature set are different, for example, the sampling rate of GPS data may be different from the update rate of road network data, even if the timestamps are the same, the intervals between data points may be different, and if there are missing data points in two data sets with the same timestamp, then the data points of the first data feature set and the second data feature set may not correspond. Therefore, in order to avoid the above situation, DTW is used to align the first data feature set and the second data feature set, which can adjust the difference in data sampling rates between the two and ensure that the time data points of the two correspond. The first data feature set and the second data feature set aligned by DTW are helpful for more effective fusion to obtain the third data feature set in the subsequent process, thereby improving the consistency and accuracy of the data; and DTW can improve the robustness of the model to time changes, so that the model can still maintain a high recognition accuracy rate;

[0065] The aligned first data feature set and the second data feature set are fused using a weighted moving average method;

[0066] For each time point, a weighted average is calculated, that is, the confidence of each feature vector in the first data feature set, each feature matrix and feature vector in the second data feature set is calculated, and the corresponding weight value is assigned according to the calculated confidence. The corresponding feature value is multiplied by the weight and then summed to obtain the fused feature value. The calculated weighted average value is used as a new feature to form a third data feature set, which will contain all the fused features. The feature vectors are weighted using the weighted moving average method, and the influence of each feature vector in the fusion process is proportional to its confidence. Specifically, for each feature vector X i , its weighted value in the third data feature set can be expressed as X i *w i , where w i is the weight of the feature vector, and the weighted feature vectors are fused to obtain the third data feature set;

[0067] The aligned first data feature set and the second data feature set are fused by weighted moving average method to integrate the information of two different data sources while retaining the key information of each feature set, so that multiple features can be comprehensively utilized to achieve complementary advantages of features and obtain more robust and accurate recognition results; the third data feature set obtained after fusion provides more comprehensive input features, which helps the model capture richer patterns and trends;

[0068] It should be noted that the weighted moving average method is a commonly used time series smoothing technology. It is a method that gives different weights to the observed values, obtains the moving average according to the different weights, and determines the predicted value based on the final moving average. A larger weight value is given to the observed values ​​close to the prediction period, and a smaller weight value is given to the observed values ​​far from the prediction period. Different weight values ​​are used to adjust the role of each observed value on the predicted value, so that the predicted value can more closely reflect the changing trend of the time series data;

[0069] As an optional implementation, please refer to Figure 3 If the recognition result is real GPS data, the real GPS data is stored in the database, which specifically includes:

[0070] Write the real GPS data into the cache queue, write the real GPS data in the cache queue into the disk file when the cache queue is full, and import the real GPS data in the disk file into the clickhouse database in batches at regular intervals;

[0071] Specifically, the GPS data acquisition operation and the data writing operation are split into two independent threads, which do not affect each other. A data cache queue is added in it. Only when the data accumulates to a set number (for example, 50,000) will it be written to the disk file in batches. After writing to the disk file, the data writing thread will periodically check whether there are data files with file names that meet the rules in the directory. If the data file is found, it will try to import the file data into the clickhouse database in batches to complete the storage of GPS data;

[0072] Please refer to Figure 2 It should be noted that the traditional way of directly writing to the database after obtaining a piece of GPS data (i.e., directly storing the received GPS data without the intermediate process of caching) is prone to data writing abnormalities or data loss during the writing process due to reasons such as data acquisition and writing speed mismatch and data errors. In the scheme of the present invention, after obtaining GPS data using the message queue SDK or Netty according to the requirements of the superior GPS data forwarding platform, the authenticity of the GPS data is identified. Considering that the clickhouse database is suitable for batch writing of data, when writing GPS data to clickhouse, it is first written to the redis cache queue. After the cache queue capacity is full, the GPS data is written to the disk file and removed from the cache queue; finally, the GPS data in the disk file format is imported into the clickhouse database in batches in an independent thread at regular intervals. Such a design allows the received GPS data to be cached in the redis cache for the first time, which not only ensures the data writing speed but also ensures that the data is not lost. When the cache queue is full, the operation of writing the disk file from the cache queue is separated from the operation of importing the disk file data into the clickhouse database, which can better avoid the influence of inconsistent reading and writing speeds on the stability of the data warehousing link;

[0073] In the present invention, in addition to clickhouse, other databases may be selected as the database ultimately used to store GPS, such as leveldb, hbase, etc., and no limitation is made here.

[0074] As an optional implementation, the method further includes converting the format of the real GPS data, converting the format of the GPS data from binary to JSON format.

[0075] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for storing and processing GPS data, characterized in that: The following steps are involved: Acquire real historical GPS data information and road network data information of the user within a first preset time period, and pre-process the historical GPS data information and the road network data information to obtain a first data feature set and a second data feature set; Fusing the first data feature set and the second data feature set to construct a third data feature set; Constructing a GPS data authenticity identification model, sending the third data feature set into the GPS data authenticity identification model for training, and obtaining a trained GPS data authenticity identification model; The GPS data of the second preset time period is identified by the trained GPS data authenticity identification model. If the identification result is authentic GPS data, the authentic GPS data is stored in a database. If the identification result is forged GPS data, it is marked.

2. A method for storing and processing GPS data according to claim 1, characterized in that: The first data feature set includes a time series feature vector, a position feature vector, and a speed feature vector.

3. The method for storing and processing GPS data according to claim 1, characterized in that: The step of obtaining the road network data information of the user within the first preset time period and preprocessing the road network data information to obtain the second data feature set specifically includes: Acquire the shape of the road the user drives on during the first preset time, and construct a road shape feature matrix based on the road shape, where the road shape includes a straight section, a curved section, an uphill section, and a downhill section; Acquire a road speed limit in a driving area of ​​the user within a first preset time, and construct a road speed limit feature vector based on the road speed limit; Acquiring traffic flow information of a driving area of ​​the user within a first preset time, and constructing a traffic flow feature vector based on the traffic flow information; The second data feature set is obtained by pre-processing the road shape feature matrix, the road speed limit feature vector, and the traffic flow feature vector and fusing the features.

4. The method for storing and processing GPS data according to claim 1, characterized in that: The step of fusing the first data feature set and the second data feature set to construct a third data feature set specifically includes: Normalizing the first data feature set and the second data feature set; Dynamic time warping is used to align the normalized first data feature set and the second data feature set, and the aligned first data feature set and the second data feature set are fused using a weighted moving average method to obtain a third data feature set.

5. A method for storing and processing GPS data according to claim 4, characterized in that: The specific steps of aligning the first data feature set and the second data feature set after normalization by using dynamic time warping are: Calculate the distance matrix between each point in the first data feature set and the second data feature set, and find an optimal path from the upper left corner to the lower right corner of the matrix; The first data feature set and the second data feature set are aligned according to the optimal path.

6. The method for storing and processing GPS data according to claim 1, characterized in that: If the recognition result is real GPS data, the real GPS data is stored in the database, which specifically includes: The real GPS data is written into a cache queue, and when the cache queue is full, the real GPS data in the cache queue is written into a disk file, and the real GPS data in the disk file is imported into a database in batches at regular intervals.

7. A method for storing and processing GPS data according to claim 6, characterized in that: It also includes converting the format of the real GPS data, converting the format of the GPS data from binary to JSON format.

Images