A method and system for incrementally capturing data of batch vehicles

Abstract

The present invention discloses a method and system for incrementally capturing data of batch vehicles, the method comprising: after receiving a vehicle storage request including a vehicle ID and vehicle information sent by a vehicle storage terminal, a server side obtains a hash key corresponding to the vehicle ID, and saves the vehicle information in an Hbase partition corresponding to the hash key; when the server side receives an incremental data capture request for batch vehicles including a data capture time period sent by a vehicle query terminal, the server side queries the hash key corresponding to the data capture time period; the server side uses the hash key and the data capture time period to capture the incremental capture data results of the batch vehicles from the Hbase partition corresponding to the hash key, and sends the captured incremental capture data results of the batch vehicles to the vehicle query terminal.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a method and system for incrementally capturing data of batch vehicles. Background Art

[0002] In order to better maintain and monitor the operation of vehicles, electric vehicles continuously send data to the data center through the on-board computer. The data includes vehicle condition data, location data, on-board battery data, etc. The daily incremental data can reach tens of millions or hundreds of millions. In order to store this data more efficiently, conveniently, and continuously, the data center will store the data in Hbase (a distributed, column-oriented open source database) after receiving the data results.

[0003] In order to avoid data hot spots and other issues, Hbase generally stores data in multiple partitions. In this case, accurate searches are usually performed through pre-partitioning or secondary indexes, and subsequent analysis usually requires data extraction and analysis by writing programs to solidify business logic. This has a great restriction on situations where the business model is not fixed and the amount of data in the incremental interval is large:

[0004] 1. The secondary index needs to maintain a set of indexes with the same data volume level;

[0005] 2. The vehicle ID needs to be generated in a linearly distributed and increasing structure. However, in actual scenarios, the vehicle ID is generally directly assigned at the factory and is not increasing.

[0006] 3. When performing batch incremental capture of all vehicles, since the vehicle ID is not fixed-incremental, there is too much data to be filtered and the processing time is too long;

[0007] 4. When switching analysis scenarios, the application program needs to be rewritten. Summary of the invention

[0008] The technical problem solved by the solution provided in the embodiment of the present invention is that when the business model is not fixed and the amount of data in the incremental interval is large, there are great constraints on the incremental capture of data for batch vehicles.

[0009] A method for incrementally capturing data of batch vehicles provided according to an embodiment of the present invention includes:

[0010] After receiving the vehicle storage request including the vehicle ID and vehicle information sent by the vehicle storage terminal, the server obtains the hash key corresponding to the vehicle ID, and saves the vehicle information to the Hbase partition corresponding to the hash key;

[0011] When the server receives an incremental data capture request for batch vehicles including a data capture time period sent by the vehicle query terminal, querying a hash key corresponding to the data capture time period;

[0012] The server side uses the hash key and the data capture time period to capture the incremental capture data results of the batch vehicles from the Hbase partition corresponding to the hash key, and sends the captured incremental capture data results of the batch vehicles to the vehicle query terminal.

[0013] A system for incrementally capturing data of batch vehicles provided according to an embodiment of the present invention includes:

[0014] The vehicle storage terminal is used to send a vehicle storage request including the vehicle ID and vehicle information to the server;

[0015] The server side is used to receive a vehicle storage request including a vehicle ID and vehicle information sent by a vehicle storage terminal, obtain a hash key corresponding to the vehicle ID, and save the vehicle information to the Hbase partition corresponding to the hash key; when receiving an incremental data capture request for batch vehicles including a data capture time period sent by a vehicle query terminal, query the hash key corresponding to the data capture time period; use the hash key and the data capture time period to capture the incremental capture data results of the batch vehicles from the Hbase partition corresponding to the hash key, and send the captured incremental capture data results of the batch vehicles to the vehicle query terminal

[0016] The vehicle query terminal is used to send an incremental capture data request for batch vehicles including a data capture time period to the server, and receive the incremental capture data results of the batch vehicles returned by the server.

[0017] According to the solution provided by the embodiment of the present invention, the present invention maintains a small number of hash keys, and forms Hbase RowKey in the form of hash key + time + vehicle ID. When performing incremental analysis, the data within the time interval is obtained in batches by circulative hash groups, and is captured by ETL means such as kettle and stored in HIVE for accurate field-level analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to understand the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0019] Figure 1 It is a flow chart of a method for incrementally capturing data of batch vehicles provided by an embodiment of the present invention;

[0020] Figure 2 is a schematic diagram of a system for incrementally capturing data of batch vehicles provided by an embodiment of the present invention;

[0021] Figure 3 This is a flow chart of a method for storing time-series big data and capturing incremental data of an Internet of Vehicles provided by an embodiment of the present invention;

[0022] Figure 4 The embodiment of the present invention provides Figure 3 Flow chart of step S2;

[0023] Figure 5 It is a schematic diagram of the processing process of a data center provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described below are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0025] Figure 1 is a flow chart of a method for incrementally capturing data of a batch of vehicles provided by an embodiment of the present invention, such as Figure 1 As shown, including:

[0026] Step S101: after receiving a vehicle storage request including a vehicle ID and vehicle information sent by a vehicle storage terminal, the server obtains a hash key corresponding to the vehicle ID, and saves the vehicle information into an Hbase partition corresponding to the hash key;

[0027] Step S102: when the server receives an incremental data capture request for batch vehicles including a data capture time period sent by the vehicle query terminal, querying a hash key corresponding to the data capture time period;

[0028] Step S103: The server side uses the hash key and the data capture time period to capture the incremental capture data results of the batch vehicles from the Hbase partition corresponding to the hash key, and sends the captured incremental capture data results of the batch vehicles to the vehicle query terminal.

[0029] The server-side database includes multiple Hbase partitions, and a hash key is configured for each Hbase partition. The maximum number of vehicle IDs to be stored and a valid time period including a start time and an end time are set for each hash key.

[0030] Specifically, after the server receives a vehicle storage request including a vehicle ID and vehicle information sent by a vehicle storage terminal, obtaining a hash key corresponding to the vehicle ID includes: after the server receives a vehicle storage request including a vehicle ID and vehicle information sent by a vehicle storage terminal, querying whether the vehicle ID has a corresponding valid hash key; when the server queries that the vehicle ID has a corresponding valid hash key, obtaining the valid hash key corresponding to the vehicle ID; when the server queries that there is no corresponding valid hash key for the vehicle ID, obtaining a valid hash key that has not reached the maximum number of vehicle IDs at present, and assigning the vehicle ID to the valid hash key that has not reached the maximum number of vehicle IDs.

[0031] Among them, the server side queries whether the vehicle computer ID has a corresponding valid hash key, including: the server side queries whether the vehicle computer ID has a corresponding hash key; when the server side queries whether the vehicle computer ID has a corresponding hash key, determines whether the hash key is within a valid time period; when it is determined that the hash key is within the valid time period, the server side determines that the vehicle computer ID has a corresponding valid hash key, otherwise, the server side determines that the vehicle computer ID has no corresponding valid hash key.

[0032] Among them, the server side queries whether the vehicle computer ID has a corresponding valid hash key, which also includes: the server side queries whether the vehicle computer ID has a corresponding hash key; when the server side queries that the vehicle computer ID has no corresponding hash key, the server side determines that the vehicle computer ID has no corresponding valid hash key.

[0033] Among them, the vehicle information includes any one or combination of the following: data collection time, vehicle status information, vehicle battery information and vehicle location information; among them, the vehicle battery information includes battery ID and battery power; the data capture time period includes data capture start time and data capture end time.

[0034] Specifically, the server side saves the vehicle information to the Hbase partition corresponding to the hash key, including: the server side obtains the data collection time from the vehicle information, and generates row key information consisting of a valid hash key + data collection time + vehicle ID; the server side saves the vehicle information to the Hbase partition corresponding to the hash key with the row key information as the primary key.

[0035] Among them, when the server side receives an incremental data capture request for batch vehicles containing a data capture time period sent by a vehicle query terminal, querying the hash key corresponding to the data capture time period includes: when the server side receives an incremental data capture request for batch vehicles containing a data capture time period sent by a vehicle query terminal, traversing the valid time period of the hash key configured for each Hbase partition in the database; when the data capture time period is within the valid time period of the hash key configured for the Hbase partition, the server side uses the hash key configured for the Hbase partition as the hash key corresponding to the data capture time period.

[0036] Among them, the server side uses the hash key and the data crawling time period to crawl the incremental crawling data results of batch vehicles from the Hbase partition corresponding to the hash key, including: the server side obtains the Hbase partition corresponding to the one or more hash keys based on the one or more hash keys corresponding to the data crawling time period; the server side crawls the incremental crawling data results of batch vehicles from the Hbase partition corresponding to the one or more hash keys based on the one or more hash keys and the data crawling time period.

[0037] Figure 2 is a schematic diagram of a system for incrementally capturing data of a batch of vehicles provided by an embodiment of the present invention, such as Figure 2 As shown, it includes: a vehicle storage terminal 201, which is used to send a vehicle storage request containing a vehicle ID and vehicle information to a server side; a server side 202, which is used to obtain a hash key corresponding to the vehicle ID after receiving the vehicle storage request containing the vehicle ID and vehicle information sent by the vehicle storage terminal, and save the vehicle information to the Hbase partition corresponding to the hash key; when receiving an incremental capture data request for batch vehicles containing a data capture time period sent by a vehicle query terminal, query the hash key corresponding to the data capture time period; using the hash key and the data capture time period, capture the incremental capture data results of the batch vehicles from the Hbase partition corresponding to the hash key, and send the captured incremental capture data results of the batch vehicles to the vehicle query terminal; a vehicle query terminal 203, which is used to send an incremental capture data request for batch vehicles containing a data capture time period to the server side, and receive the incremental capture data results of the batch vehicles returned by the server side.

[0038] That is to say, the vehicle storage terminal sends data to the server at a frequency of 30 seconds per piece, and the data format is as follows:

[0039]

[0040] After receiving and processing the data, the server stores the data in Hbase; the vehicle query terminal sends a query request to the server, where the query request contains the time interval of the data start time and the data end time. The server obtains data according to the time interval and returns it to the vehicle query terminal.

[0041] Figure 3 This is a flow chart of a method for storing time-series big data and capturing incremental data of an Internet of Vehicles provided by an embodiment of the present invention, which includes:

[0042] Step S1, design the maximum number of vehicle computers that can be stored in each partition of Hbase storage;

[0043] Step S2, maintaining the vehicle ID destination hash key, whose structure is: (vehicle ID: belonging hash key value), this belonging relationship can be automatically maintained;

[0044] Further, such as Figure 4 As shown, the specific process of step S2 is as follows:

[0045] Step S21: Maintain a corresponding relationship table between the vehicle ID and the hash key. The table structure is as follows:

[0046] Car ID Hash key value AAAA1 A001 AA1A2 A001 …… …… AA01B A002

[0047] Step S22: Get the vehicle ID and query the corresponding hash key value. If it is a new vehicle ID or an unmaintained vehicle ID, assign it a hash key whose vehicle skill number is not full.

[0048] Step S23: Maintain the hash key newly assigned to the vehicle ID into the relationship table.

[0049] Step S3: When storing, obtain the corresponding hash key, such as 100, through the vehicle ID of the uploaded message, and generate the corresponding HBASE RowKey: 100_data time_vehicle ID;

[0050] Step S4: Determine the incremental data in a partition by (hash key_start time, hash key_end time), and then obtain all hash keys in a loop or multi-threaded manner to obtain the full amount of incremental data in the interval;

[0051] Step S5: Obtain all incremental data by assembling the incremental data of all partitions for analysis or storage in the ODS layer of the data warehouse.

[0052] Figure 5 Schematic diagram of the processing process of a data center provided by an embodiment of the present invention. Figure 5 As shown, including:

[0053] Step 501: Car Tbox (equivalent to Figure 2 The vehicle storage terminal in the system will collect vehicle data on a regular basis (every 30 seconds), including:

[0054] TBoxID (vehicle ID number), data collection time (date and time accurate to seconds, such as: 20210716122030), vehicle status data, vehicle operation data, battery operation data, etc.

[0055] Step 502: After assembling according to the data protocol, send it to the gateway and transfer it to the data center (equivalent to Figure 2 On the server side) in Kafka's Topoc;

[0056] The real-time collection ETL subscribes to the Kafka topic, obtains the reported data stream in real time, parses the data according to the protocol, and obtains the vehicle ID, data collection time, etc.

[0057] Get the hash key corresponding to the vehicle ID according to the configuration table. If the vehicle ID is in the existing configuration table and no key is assigned, randomly select a key from the unfilled group and assign it to the vehicle ID (if all keys are full, generate a new key), and store the relationship between the vehicle ID and the key in the configuration table.

[0058] The configuration table of vehicle ID and hash key is as follows:

[0059] Car ID Key Effective time Expiration time 45637902 A001 20210521132500 20991231235959 44802820 A002 20210527143830 20991231235959 … … … …

[0060] The definition of the key and the number of vehicles contained in each key can be defined according to the cluster host configuration.

[0061] The key + data collection time + vehicle ID form the Rowkey in Hbase. The Rowkey is concatenated with the collected data and written into Hbase in real time.

[0062] Step 503: The incremental extraction program has two parameters: start time and end time, which can be set as required, such as:

[0063] 1. In the T+1 timed increment program, the starting time is generally 0:00 yesterday and the ending time is 23:59:59 yesterday.

[0064] 2. When you need to query incremental data directly from Hbase, you usually set the start time and end time according to specific business requirements. For example, if you want to obtain vehicle data within 2 hours, the start time is two hours before the current time, and the end time is the current time.

[0065] Get all hash keys from the configuration table and allocate the corresponding number of threads according to demand. The concatenated hash key + start time of each thread is used as the start rowkey, and the hash key + end time is used as the end rowkey.

[0066] Because Hbase stores data in partitions according to hash keys, data hot spots are avoided. In multi-threaded situations, data in the incremental interval can be obtained with high efficiency, written to the data warehouse, or returned to the specified location through the API.

[0067] The row key in this application is like the primary key in the database, which is used to uniquely identify a record. The data in the Base table is stored in different regions according to the RowKey.

[0068] Hash key: Use the rules provided in the article to formulate rowkeys, evenly distribute data within a certain period of time to each region partition to avoid hot spots.

[0069] The role of hash keys: 1. Avoid hotspot writes; 2. When querying a continuous time interval, you can obtain the hash keys involved in this time interval, and simply process multiple threads and loops to combine them into rowkeys to obtain data in HBase:

[0070] In addition, in this application, you need to first find the valid key within the time range (such as: 20211101120101, 20211102120101):

[0071] Car ID Key Effective time Expiration time 45637902 A001 20210521132500 20991231235959 44802820 A002 20210527143830 20991231235959 … … … …

[0072] If you get: A001, A002,

[0073] Combine the query RowKey conditions and obtain the values ​​of the following conditions through loops and multithreading:

[0074] rowkey>=A001_20211101120101and rowkey<=A001_20211102120101

[0075] rowkey>=A002_20211101120101and rowkey<=A002_20211102120101

[0076] According to the solution provided in the embodiment of the present invention, the maintenance cost of the secondary index can be reduced, and the storage space can be reduced; the data extraction efficiency of the ODS layer can be effectively improved, and the time consumption of incremental operations can be reduced.

[0077] Although the present invention is described in detail above, the present invention is not limited thereto, and those skilled in the art can make various modifications based on the principles of the present invention. Therefore, all modifications made based on the principles of the present invention should be understood to fall within the protection scope of the present invention.

Claims

1. A method for incrementally capturing data of batch vehicles, characterized in that: include: After receiving the vehicle storage request including the vehicle ID and vehicle information sent by the vehicle storage terminal, the server obtains the hash key corresponding to the vehicle ID, and saves the vehicle information to the Hbase partition corresponding to the hash key; It includes: After receiving the vehicle storage request including the vehicle ID and vehicle information sent by the vehicle storage terminal, the server side queries whether the vehicle ID has a corresponding valid hash key; When the server queries that the vehicle computer ID has a corresponding valid hash key, the server obtains the valid hash key corresponding to the vehicle computer ID; When the server queries the vehicle computer ID and finds that there is no corresponding valid hash key, obtain the valid hash keys that do not currently reach the maximum number of vehicle computer IDs, and assign the vehicle computer ID to the valid hash keys that do not currently reach the maximum number of vehicle computer IDs; When the server receives an incremental data capture request for batch vehicles including a data capture time period sent by the vehicle query terminal, querying a hash key corresponding to the data capture time period; The server side uses the hash key and the data capture time period to capture the incremental capture data results of the batch vehicles from the Hbase partition corresponding to the hash key, and sends the captured incremental capture data results of the batch vehicles to the vehicle query terminal.

2. The method according to claim 1, characterized in that The database on the server side includes multiple Hbase partitions, and a hash key is configured for each Hbase partition. The maximum number of vehicle machine IDs to be stored and a valid time period including a start time and an end time are set for each hash key.

3. The method according to claim 1, characterized in that The server side queries whether the vehicle computer ID has a corresponding valid hash key, including: The server queries whether the vehicle ID has a corresponding hash key; When the server queries the vehicle ID and finds that it has a corresponding hash key, determining whether the hash key is within a valid time period; When it is determined that the hash key is within the valid time period, the server determines that the vehicle ID has a corresponding valid hash key; otherwise, the server determines that the vehicle ID has no corresponding valid hash key.

4. The method according to claim 1, characterized in that: The server side queries whether the vehicle computer ID has a corresponding valid hash key and further includes: The server queries whether the vehicle ID has a corresponding hash key; When the server side inquires that there is no corresponding hash key for the vehicle computer ID, the server side determines that there is no corresponding valid hash key for the vehicle computer ID.

5. The method according to any one of claims 1 to 4, characterized in that: The vehicle information includes any one or combination of the following: data collection time, vehicle status information, vehicle battery information and vehicle location information; wherein the vehicle battery information includes battery ID and battery power; the data capture time period includes data capture start time and data capture end time.

6. The method according to claim 1, characterized in that The server side saves the vehicle information to the Hbase partition corresponding to the hash key, including: The server obtains the data collection time from the vehicle information and generates row key information consisting of a valid hash key + data collection time + vehicle ID; The server side saves the vehicle information into the Hbase partition corresponding to the hash key using the row key information as the primary key.

7. The method according to claim 6, characterized in that When the server receives an incremental data capture request for batch vehicles including a data capture time period sent by the vehicle query terminal, querying a hash key corresponding to the data capture time period includes: When the server receives an incremental data capture request for batch vehicles including a data capture time period sent by the vehicle query terminal, it traverses the valid time period of the hash key configured for each Hbase partition in the database; When the data capture time period is within the valid time period of the hash key configured for the Hbase partition, the server uses the hash key configured for the Hbase partition as the hash key corresponding to the data capture time period.

8. The method according to claim 7, characterized in that The server side uses the hash key and the data capture time period to capture the incremental capture data of batch vehicles from the Hbase partition corresponding to the hash key. The result includes: The server side obtains the Hbase partitions corresponding to the one or more hash keys according to the one or more hash keys corresponding to the data capture time period; The server side uses the one or more hash keys and the data capture time period to capture incremental capture data results of batch vehicles from the Hbase partitions corresponding to the one or more hash keys.

9. A system for incrementally capturing data of batch vehicles, characterized in that: include: The vehicle storage terminal is used to send a vehicle storage request including the vehicle ID and vehicle information to the server; The server side is used to obtain the hash key corresponding to the vehicle ID after receiving the vehicle storage request including the vehicle ID and vehicle information sent by the vehicle storage terminal, and save the vehicle information to the Hbase partition corresponding to the hash key; when receiving the incremental capture data request for batch vehicles including the data capture time period sent by the vehicle query terminal, query the hash key corresponding to the data capture time period; use the hash key and the data capture time period to capture the incremental capture data results of the batch vehicles from the Hbase partition corresponding to the hash key, and send the captured incremental capture data results of the batch vehicles to the vehicle query terminal; The vehicle query terminal is used to send a request for incremental capture data of batch vehicles including a data capture time period to the server, and receive the incremental capture data results of batch vehicles returned by the server; The server is further configured to receive a vehicle storage request including a vehicle ID and vehicle information from a vehicle storage terminal, query whether the vehicle ID has a corresponding valid hash key; and when the vehicle ID has a corresponding valid hash key, obtain the valid hash key corresponding to the vehicle ID; When there is no corresponding valid hash key for the vehicle computer ID, obtain the valid hash keys that do not currently reach the maximum number of vehicle computer IDs, and assign the vehicle computer ID to the valid hash keys that do not currently reach the maximum number of vehicle computer IDs.

Images