Method, computer readable medium and distributed system for storing and providing georeferenced vehicle data
By using machine learning methods in a distributed system to predict the scaling level of georeferenced vehicle data and determine the partitioning scheme of time partition and location partition, the problem of low storage and query efficiency of georeferenced vehicle data in the prior art is solved, and the implementation of the realization of the realization of georeferenced vehicle data Efficient and fault-tolerant data storage and query.
Patent Information
- Application Number
- CN202380047287.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-07-15
- Filing Date
- 2023-04-20
- Publication Date
- 2025-02-14
AI Technical Summary
The prior art has problems of uneven data distribution and inefficient storage when storing and querying georeferenced vehicle data, resulting in low data query efficiency.
By using machine learning methods in a distributed system to predict the scaling level of the location information of the georeferenced vehicle data, and determine the partitioning scheme of time partition and location partition based on the time information and location information, efficiently store and query the georeferenced vehicle data. .
It realizes efficient storage and query of georeferenced vehicle data, ensures the fault tolerance of data and the error-free query, and improves the efficiency of data processing and query.
Smart Images
Figure CN119452355A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method for storing and providing geo-referenced vehicle data. The invention also relates to a computer-readable medium for storing and providing geo-referenced vehicle data and a distributed system for storing and providing geo-referenced vehicle data. Background Art
[0002] Current vehicles may transmit various vehicle data to a backend server. The vehicle data is typically a large data set, the size of which may include several GB of data. Partitioning methods are known from the prior art for partitioning large data sets in order to be able to query the data more efficiently. Known software solutions include, for example, Apache Spark or Geospark. However, the known software solutions have the disadvantage that the data must be distributed between the computing units of the distributed system before they can be processed on them. Furthermore, a disadvantage of the known methods is that each computing unit creates its own partitioning scheme, which leads to inefficient storage and inefficient querying of the stored data. Summary of the invention
[0003] Therefore, it is an object of the present invention to efficiently improve the storage and provision of geo-referenced vehicle data. In particular, it is an object of the present invention to enable fault-tolerant storage of geo-referenced vehicle data, which efficiently enables error-free querying of the geo-referenced vehicle data.
[0004] This object is achieved by the features of the independent claim. Advantageous embodiments and further developments of the invention are disclosed in the dependent claims.
[0005] According to a first aspect, the invention features a method for storing and providing geo-referenced vehicle data. The method may be a computer-implemented method. Preferably, the geo-referenced vehicle data is stored and provided on a distributed system. The distributed system may be, for example, a cloud computing system, a grid computing system, or a cluster system.
[0006] The method includes receiving geo-referenced vehicle data via a data memory of a distributed system. For example, the geo-referenced data may be received from a plurality of vehicles in a fleet. The method also includes determining a subset of the geo-referenced vehicle data via a first computing unit of the distributed system, the subset including data with time information and location information of the geo-referenced vehicle data, and ascertaining a partitioning scheme including time partitioning and location partitioning via the first computing unit of the distributed system based on the time information and location information of the subset of the geo-referenced vehicle data. The method trains a machine learning method via the first computing unit of the distributed system with the ascertained partitioning scheme and the subset of the geo-referenced vehicle data for predicting a zoom level of location information of the geo-referenced vehicle data, and partitions the received data of the geo-referenced vehicle data using the trained machine learning method via a plurality of computing units of the distributed system. The zoom level may include a division of a world map into respective predetermined areas of a predetermined size. The partitioned data of the received geo-referenced vehicle data are stored in a data memory of the distributed system via the plurality of computing units of the distributed system. The method finally comprises: receiving a query message for geo-referenced vehicle data, the query message comprising time information and location information; and providing the geo-referenced vehicle data of the query message, wherein the geo-referenced vehicle data is provided using the stored partitioned data.
[0007] Advantageously, the method enables efficient storage and querying of geo-referenced vehicle data.
[0008] According to an advantageous embodiment of the invention, the size of the subset of georeferenced data can be selected such that a partitioning scheme can be ascertained for the subset of georeferenced data by the first computing unit of the distributed system and / or the machine learning method can be trained by the first computing unit of the distributed system. The machine learning method can thus be trained using a partitioning scheme that is complete for the subset.
[0009] According to another advantageous embodiment, the subset of the georeferenced data can be determined based on a time data progression, wherein the time data progression preferably represents a day, a week and / or a month; and / or wherein the time data progression preferably represents the difference between working days, i.e. Monday to Friday, and weekends, i.e. Saturday and Sunday. This makes it possible to efficiently ascertain a representative subset of the georeferenced data.
[0010] According to another advantageous embodiment, the partitioning scheme can be determined in a quadtree, thereby making it possible to efficiently calculate the partitioning scheme on a single computing unit of a distributed system.
[0011] According to another advantageous embodiment, the partitioning scheme can identify time partitions and location partitions for the time information and location information of the data set of the subset of geo-referenced data, wherein the location partitions are preferably uniquely determined by a Geohash, and wherein the Geohash preferably specifies a zoom level of the location partitions. This enables the specification of a partitioning scheme that can efficiently store and query geo-referenced vehicle data.
[0012] According to another advantageous embodiment, training the machine learning method can include ascertaining hyperparameters of the machine learning method by means of cross validation. This makes it possible to efficiently improve the training of the machine learning method.
[0013] According to another advantageous embodiment, the method may further include: distributing the trained machine learning method from the first computing unit of the distributed system to the plurality of computing units of the distributed system, whereby the trained machine learning method may be used on the plurality of computing units to divide and store the geo-referenced data into partitions.
[0014] According to another advantageous embodiment, partitioning the received geo-referenced vehicle data by the plurality of computing units of the distributed system using a trained machine learning method may include: determining the time partition of the partitioning scheme according to the time information of the data set; predicting the zoom level of the location partition according to the location information of the received geo-referenced vehicle data data set by one of the plurality of computing units using a trained machine learning method; determining the Geohash by the computing unit in the plurality of computing units using the location information and the predicted zoom level of the location partition; and determining the location partition of the partitioning scheme according to the Geohash. This enables the data set of geo-referenced data to be efficiently divided into partitions with different zoom levels. In addition, the data set of geo-referenced vehicle data can be stored in partitions with different zoom levels in a fault-tolerant manner.
[0015] According to another advantageous embodiment, storing the partitioned data of the received geo-referenced vehicle data in a data memory of the distributed system by the plurality of computing units of the distributed system may comprise storing the data sets of the received geo-referenced vehicle data in the determined time and location partitions. This enables efficient storage of the geo-referenced vehicle data.
[0016] According to another advantageous embodiment, providing the geo-referenced vehicle data of the query message may include: determining one or more time partitions based on the time information of the query message; ascertaining the Geohash of all zoom levels based on the location information of the query message; determining the location partition of the ascertained Geohash; querying the geo-referenced vehicle data of the time information and location information of the query message in the determined one or more time partitions and the determined location partition; and providing the queried geo-referenced vehicle data in response to the query message. This enables efficient querying of vehicle data. By determining the Geohash of all relevant zoom levels of the location information, the geo-referenced vehicle data can be correctly found and queried when querying for situations where the trained machine learning method predicts an incorrect zoom level. In other words, even if the geo-referenced vehicle data is stored in an incorrect partition, the query can provide correct geo-referenced vehicle data.
[0017] According to another aspect, the invention features a computer-readable medium for storing and providing geo-referenced vehicle data, the computer-readable medium including instructions that, when executed on one or more computing units of a distributed system, perform the method described above.
[0018] According to another aspect, the invention features a distributed system for storing and providing geo-referenced vehicle data, which is designed to carry out the method described above.
[0019] Further features of the invention are derived from the claims, the drawings and the description of the drawings. All features and combinations of features mentioned above in the description and the features and combinations of features mentioned below in the description of the drawings and / or shown individually in the drawings can be used not only in the respectively indicated combination but also in other combinations or alone. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. Further details, preferred embodiments and further improvements of the present invention are derived from this. Schematically:
[0021] Figure 1 An exemplary method for storing and querying geo-referenced vehicle data in a distributed system is shown. DETAILED DESCRIPTION
[0022] Figure 1An exemplary method 100 for storing and querying geo-referenced vehicle data in a distributed system is shown in detail. Vehicle data generally includes time data and geo-referenced data. For example, geo-referenced data may include latitude and longitude coordinates of the vehicle's location. In addition, vehicle data may include other application-specific data. Vehicle data, in particular geo-referenced vehicle data, can be used for location-based services, map-based functions and / or driver assistance systems of the vehicle. Geo-referenced vehicle data can be a large data set with hundreds of Gigabytes to several Terabytes of data. Geo-referenced vehicle data can be saved with the help of partitions by time and location. Partitioning by location can be performed, for example, using Geohash. In order to achieve partitioning with Geohash, the world map can be divided into different geometric areas (e.g., rectangles or hexagons) and into different zoom levels. A zoom level can specify the size of a geometric area. It is applicable that each Geohash of a zoom level n is fully filled with each Geohash of zoom level n+1 in the same geometric area. For example, to encode the Geohash, a combination of one or more letters and / or one or more numbers can be defined, so that each area of the Geohash can be uniquely identified. The zoom level can specify the size of the partition by the size of the geometric area of the zoom level. By specifying the size of the partition, the amount of data that is classified into the partition from the total amount of georeferenced vehicle data can be determined.
[0023] In order to be able to define a plurality of zoom levels, preferably all georeferenced vehicle data must be taken into account. Due to the size of the data set of georeferenced vehicle data, only a subset of the georeferenced vehicle data may be used to define the zoom levels. Furthermore, a subset of the georeferenced vehicle data may include a sub-set of attributes of all or some subsets of the georeferenced vehicle data. In order to be able to distribute the georeferenced vehicle data into partitions as efficiently as possible, method 100 proposes to use a machine learning method to partition the georeferenced vehicle data.
[0024] In detail, the method 100 may receive 102 georeferenced vehicle data via a data memory of the distributed system. The georeferenced vehicle data may be received from a vehicle or a plurality of vehicles (e.g. a fleet of vehicles or a subset of a fleet of vehicles). The method 100 may determine 104 a subset of the georeferenced vehicle data by a first computing unit of the distributed system. The subset may include data of the georeferenced vehicle data with time information (e.g. date) and location information (e.g. position of the vehicle). The subset preferably does not contain application-specific data. This allows the size of the georeferenced vehicle data to be efficiently reduced. The size of the subset of the georeferenced vehicle data is preferably selected such that the subset can be used on a single computer of the distributed system to train the machine learning method. For example, the size of the subset of the georeferenced vehicle data may be selected such that regulations regarding memory consumption and computing time are met when training the machine learning method. Preferably, the size of the subset of the georeferenced vehicle data comprises a size of several Gigabytes of data. The subset of the georeferenced vehicle data is further selected such that the distribution of the data of the subset corresponds to the distribution of all georeferenced vehicle data with respect to time information and / or location information. This may avoid errors when training the machine learning method. For example, the subset of geo-referenced vehicle data may only include data about time information and location information. Application-specific data of the geo-referenced vehicle data, which are not relevant for the training of the machine learning method, may no longer be taken into account for the determination of the subset of geo-referenced vehicle data. In addition, for example, only the date of the time information may be used for the determination of the subset. The timestamp of the time information may no longer be taken into account for the determination of the subset.
[0025] Method 100 can determine 106 a partitioning scheme including a time partition and a location partition by a first computing unit of a distributed system based on the time information and the location information of the subset of the geo-referenced vehicle data. For example, method 100 can determine a partitioning scheme, in particular an optimal partitioning scheme, for the subset of the geo-referenced vehicle data using one or more known methods. A known method for calculating a partitioning scheme can determine an optimal partitioning scheme for the subset of the geo-referenced vehicle data, for example, using a quadtree. The determined partitions of the partitioning scheme can be used as labels for a multi-class classification problem that is to be solved with the aid of a machine learning method.
[0026] Method 100 can train 108 a machine learning method for predicting the zoom level of location information of geo-referenced vehicle data using the determined partitioning scheme and the subset of geo-referenced vehicle data by a first computing unit of the distributed system. The machine learning method can be a known machine learning method. Various machine learning methods can be evaluated with expert knowledge or methods such as cross-validation to determine a suitable machine learning method. The purpose of training the machine learning method is to predict the zoom level of the location information for each time information (in particular, each new date) and each location information (such as latitude and longitude coordinates). The Geohash can then be determined for the predicted zoom level. For this purpose, the labels of the determined partitions can be used as outputs of the machine learning method. The labels can correspond to the zoom levels of the location information. In order to train the machine learning method, the subset of geo-referenced vehicle data can be used as input.
[0027] The method 100 can partition 110 the data of the received geo-referenced vehicle data by multiple computing units of a distributed system using a trained machine learning method. For each data set of the received geo-referenced vehicle data, time information and location information are used as input to the trained machine learning method to predict the zoom level. This is done similarly to the training of the machine learning method, as described above. Using the zoom level and location information, a Geohash associated with this can be calculated. The location partition can be uniquely determined using the Geohash. In addition, the time partition can be determined using time information, for example using a timestamp for each data set of the geo-referenced vehicle data.
[0028] The trained machine learning method can perform a probability-based prediction of the zoom level for each date of the data set of geo-referenced vehicle data. This may result in that the optimal zoom level for each date is not always predicted by the machine learning method, and the data set of geo-referenced vehicle data is not stored in the optimal partition. By storing the geo-referenced vehicle data in a fault-tolerant storage scheme, subsequent queries on the geo-referenced vehicle data can be performed without loss. The fault-tolerant storage scheme can be implemented by partitioning the geo-referenced vehicle data using Geohash. For Geohash, it is applicable that the Geohash of zoom level n is mapped to a set of Geohash of zoom level n+1. In addition, for Geohash, it is applicable that all Geohash of zoom level n+1 is completely contained in and fills the Geohash of zoom level n. When querying geo-referenced vehicle data partitioned by Geohash, all relevant Geohash of all zoom levels can be queried, and therefore, the data set of geo-referenced vehicle data that has been stored in another zoom level can also be found. Therefore, it is possible to query the geo-referenced vehicle data without loss.
[0029] The method 100 may store 112 the partitioned data of the received geo-referenced vehicle data in a data memory of the distributed system by the plurality of computing units of the distributed system. In the case of using time partitioning and location partitioning, each data set of geo-referenced vehicle data may be processed and stored independently of each other by one of the plurality of computing units of the distributed system. An example of a storage scheme for storing data sets of geo-referenced vehicle data may be constructed as follows:
[0030] Root folder /
[0031] - Date = 2022-03-07
[0032] -Geohash=g12
[0033] All datasets on March 7, 2022 with Geohash=g12
[0034] -Geohash=g1234
[0035] All datasets with Geohash=g1234 on March 7, 2022
[0036] -Geohash=f567
[0037] All datasets with Geohash=f567 on March 7, 2022
[0038] - Date = 2022-03-08
[0039] -Geohash=g12
[0040] All datasets with Geohash=g12 on March 8, 2022
[0041] -Geohash=g1234
[0042] All datasets on March 8, 2022 with Geohash=g1234
[0043] -Geohash=f567
[0044] All datasets with Geohash=f567 on March 8, 2022
[0045] -…
[0046] The method 100 may receive 114 a query message for geo-referenced vehicle data, the query message including time information and location information. In addition, the method 100 may provide 116 geo-referenced vehicle data to the query message, wherein the geo-referenced vehicle data is provided using the stored partitioned data. An exemplary query message includes a query for all data of geo-referenced vehicle data for a specific location as location information and from a specific time interval as time information. The specific time interval of the exemplary query message may include March 7, 2022 and March 8, 2022. In an exemplary planning scheme, the days of the time interval may be directly mapped to the time information of the time partition. The specific location of the exemplary query message may be Munich. For the specific location Munich, the method may determine the relevant Geohash for all zoom levels. For example, the specific location Munich may include a Geohash g12 for the first zoom level of the specific location and a Geohash g1234 for the second zoom level of the specific location. Geohash may be directly mapped to the partition of the storage scheme. The results of the query include all data of the geo-referenced vehicle data stored in the identified partition of the storage scheme.
[0047] The machine learning method can be retrained for changes in the distribution of geo-referenced vehicle data. For this purpose, as described above, new training data can be ascertained and the machine learning method can be trained with the new training data.
[0048] Advantageously, the method may efficiently ascertain a partitioning scheme for geo-referenced vehicle data, which enables error-tolerant storage of the geo-referenced vehicle data. Reference numerals list
[0049] 100 Methods
[0050] 102 Receive georeferenced vehicle data
[0051] 104 Determine a subset of geo-referenced vehicle data
[0052] 106 Find out the partition plan
[0053] 108 Training Machine Learning Methods
[0054] 110 Partitioning of geo-referenced vehicle data
[0055] 112Storing partitioned data
[0056] 114 Receive query message
[0057] 116 provides geo-referenced vehicle data.
Claims
1. A method for storing and providing geo-referenced vehicle data, the method comprising: receiving geo-referenced vehicle data via a data store of the distributed system; determining, by a first computing unit of the distributed system, a subset of the geo-referenced vehicle data, the subset comprising data having time information and location information of the geo-referenced vehicle data; determining, by a first computing unit of the distributed system, a partitioning scheme including a time partition and a location partition based on the time information and the location information of the subset of the geo-referenced vehicle data; training a machine learning method by a first computing unit of the distributed system with the ascertained partitioning scheme and the subset of the geo-referenced vehicle data for predicting a zoom level of the location information of the geo-referenced vehicle data; partitioning the data of the received geo-referenced vehicle data by a plurality of computing units of the distributed system using a trained machine learning method; storing, by the plurality of computing units of the distributed system, partitioned data of the received geo-referenced vehicle data in a data memory of the distributed system; receiving a query message for geo-referenced vehicle data, the query message including time information and location information; and Geo-referenced vehicle data of the query message are provided, wherein the geo-referenced vehicle data are provided using stored partitioned data.
2. The method according to claim 1, wherein: The size of the subset of geo-referenced data is selected such that a partitioning scheme can be ascertained for the subset of geo-referenced data by the first computing unit of the distributed system and / or the machine learning method can be trained by the first computing unit of the distributed system.
3. The method according to any one of the preceding claims, wherein: determining said subset of geo-referenced data according to a temporal data progression; and Wherein, the time data progression preferably represents a day, a week and / or a month; and / or Therein, the time data progression preferably represents the difference between weekdays, ie Monday to Friday, and weekends, ie Saturday and Sunday.
4. A method according to any one of the preceding claims, wherein: The partitioning scheme is found using a quadtree.
5. A method according to any one of the preceding claims, wherein: The partitioning scheme identifies time partitions and location partitions for the time information and location information of the dataset of the subset of geo-referenced data, Wherein, the location partition is preferably uniquely determined by Geohash, and The Geohash preferably specifies the zoom level of the location partition.
6. A method according to any one of the preceding claims, wherein: Training the machine learning method comprises ascertaining hyperparameters of the machine learning method by means of cross validation.
7. The method according to any one of the preceding claims, further comprising: The trained machine learning method is distributed from the first computing unit of the distributed system to the plurality of computing units of the distributed system.
8. A method according to any one of the preceding claims, wherein: Partitioning the data of the received geo-referenced vehicle data by the plurality of computing units of the distributed system using a trained machine learning method comprises: Determine the time partition of the partition scheme based on the time information of the data set; predicting, by one of the plurality of computing units, a zoom level of a location partition based on location information of the received dataset of geo-referenced vehicle data using a trained machine learning method; determining, by the one of the plurality of computing units, a Geohash using the location information and the predicted zoom level of the location partition; and Determine the location of the zoning scheme based on Geohash.
9. A method according to any one of the preceding claims, wherein: Storing the partitioned data of the received geo-referenced vehicle data in a data memory of the distributed system by the plurality of computing units of the distributed system comprises: The data sets of the received geo-referenced vehicle data are stored in the determined time and location partitions.
10. A method according to any one of the preceding claims, wherein: The vehicle data providing the geo-reference of the query message includes: Determine one or more time partitions according to the time information of the query message; Find the Geohash of all zoom levels based on the location information of the query message; Determine the locational subdivision of the identified Geohash; querying the geo-referenced vehicle data for the time information and the location information of the query message in the determined one or more time partitions and the determined location partitions; and The queried geo-referenced vehicle data is provided in response to the query message.
11. A computer readable medium for storing and providing geo-referenced vehicle data, wherein: The computer readable medium comprises instructions which, when executed on one or more computing units of a distributed system, perform the method according to any one of claims 1 to 10.
12. A distributed system for storing and providing geo-referenced vehicle data, wherein: The distributed system is designed to carry out the method according to any one of claims 1 to 10 .