Pre-partitioning method based on Internet of Vehicles Hbase time series data

Abstract

The invention relates to a pre-partitioning method based on Internet of Vehicles Hbase time series data, and the method comprises the following steps: S1, defining a RowKey generation rule algorithm of the Hbase time series data, obtaining a RowKey value of a piece of data reported by a vehicle with a fixed length of 23 bits, and taking the first 8 bits of the RowKey value as spitKey values of pre-partitioning; S2, creating a pre-partition according to the splitKey value; and S3, distributing a partition for the newly-added vehicle according to the splitKey value, and when the partition corresponding to the splitKey value of the newly-added vehicle does not exist, creating a new partition according to the splitKey value and writing the data of the newly-added vehicle into the new partition. On one hand, resource allocation balance of each storage node can be ensured, and on the other hand, data storage space can be saved.

Description

technical field [0001] The present invention relates to the field of the Internet of Vehicles, in particular to a pre-partitioning method based on Hbase time series data of the Internet of Vehicles. Background technique [0002] With the rapid development of the Internet of Vehicles industry, the number of vehicles that the Internet of Vehicles service platform is facing has been upgraded from thousands to tens of thousands, millions, and tens of millions of vehicles. The Hbase time series data storage mode is used to store the history of the Internet of Vehicles service platform Trajectory data is fairly common. The storage capacity of time series data and efficient, convenient, and sustainable operation management have become indispensable indicators in the system architecture. As for the storage capacity of data, the horizontal expansion of storage nodes can be realized by means of distributed load balancing at the system architecture level, so as to achieve unlimited im...

AI Technical Summary

Problems solved by technology

[0004] ① The total number of vehicles in the project must be fixed in order to obtain the splitKey value of the pre-partition, and with the operation of the project, it is inevitable to increase the number of vehicles;

[0005] ② When new vehicles are added, since the splitKey value of the obtained pre-partition is sorted by the vehicle id string, it is not convenient to specify and store the data of this batch of new vehicles by adding a new partition; and the newly added vehicles are sorted by characters The string sorting will be scattered on the partitions that have been created and allocated. Some partitions are under too much pressure, causing uneven resource allocation of each node in hbase, resulting in performance degradation of some nodes in the cluster, and affecting the normal operation of the business;

[0006] ③ After adding new vehicles, if the originally planned cluster size cannot bear the increased pressure on the cluster due to the addition of new vehicles, and new storage nodes need to be expanded to share the pressure on the cluster, the partitions of the nodes with high pressure will be migrated to the newly added nodes In order to reduce the pressure on the nodes, when a large number of partitions are migrated to the newly added storage nodes, in order to meet the data localization of hbase, these migrated partitions need to migrate the data managed by them to the newly added storage nodes. Such an operation Increase the burden of disk I / O and network communication between clusters, affecting the normal operation of the business;

[0007] ④ The data of the general vehicle network project needs to be kept for at least 2 to 3 years. The RowKey generation rule of "vehicle id + _ + data reporting time stamp (accurate to milliseconds) + _ + three-digit random number" is adopted, As a result, all the data of each car is stored in one partition, and as time goes by, the amount of data stored in each partition will increase, which will seriously affect the query performance of the partition; and for more than 2 to 3 years When cleaning expired historical data, because hbase deletes data only marks the data as deleted, it is not a real deletion. Only when performing a large merge on the partition will the expired data be deleted from the disk. A large number of partitions perform a large merge Sometimes it will cause disk I / O between clusters, network communication will be affected, and the normal operation of the business will be affected;

[0008] To sum up, although the traditional pre-partitioning method can maintain effective storage management for a certain period of time, the long-term operation and management will cause a serious burden on the cluster

Images