A real-time aggregation method for vehicle data

Abstract

The present invention discloses a real-time aggregation method for vehicle data. The data source is consumed from a kafka cluster to ensure the real-time performance of the data source. Then, the data results are summarized based on the kafka unbounded flow window. After the data source is obtained, grouping operations and aggregation are performed based on the real-time operation summary. The data summary can reach 20 dimensions of calculation, and the results are cached to redis storage respectively, so that the operation results can be continuously written to the redis cache, and redis is used as a cache database that can be written in an infinite loop, and then read and updated to the relational database by a scheduled task. The present invention can realize the aggregation and update of the latest data, and the total time of the entire operation and writing cache to persistence is controlled within a delay of less than 10 minutes; compared with T+1 aggregation, the real-time performance of the data is higher, the operation results are continuously updated, and the efficiency is higher.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle networking and relates to a real-time aggregation method for vehicle data. Background Art

[0002] With the development of Internet of Vehicles technology, vehicle information needs to be sent to the backend. For example, in the real-time mileage aggregation of vehicles, each vehicle reports to the gateway every 5 seconds, and then consumes the vehicle reported data from the gateway and stores it. However, as the number of vehicles that need to be processed by the backend increases, it becomes increasingly difficult to consume directly into a relational database. Generally, it is consumed first to the middleware (such as HDFS) for storage, and after T+1 time (that is, only the data summary of the previous day can be calculated on the same day), the summary result is obtained, and then written to the relational database for storage, which cannot guarantee the real-time performance of vehicle information. Summary of the invention

[0003] The technical problem solved by the present invention is to provide a real-time aggregation method for vehicle data to achieve real-time aggregation of vehicle information.

[0004] The present invention is achieved through the following technical solutions:

[0005] A real-time vehicle data aggregation method includes the following operations:

[0006] 1) Real-time vehicle data is sent to the gateway through the vehicle terminal, and the gateway sends it to Kafka. The real-time vehicle data is saved as a binary unbounded stream;

[0007] 2) The data of the binary unbounded stream is then received by the parser, which parses it into plain text according to the prescribed business method;

[0008] 3) The grouping operator receives the plain text and groups it according to the data number and data time; and calculates the grouped data in groups according to the business method to obtain the vehicle intermediate data including the data number, data time, timestamp and business content;

[0009] 4) Cache the intermediate vehicle data to Redis;

[0010] 5) The summary calculator periodically extracts the intermediate data of the vehicle from the Redis cache, performs calculations according to the business method, writes the calculation results to the database, and updates the vehicle business status.

[0011] When the intermediate data of the vehicle is cached in Redis, the initial business status of the vehicle is saved in the Redis persistent area to prevent the loss of memory data; when an exception occurs, the initial business status data of the vehicle will be loaded into the next computing task.

[0012] The Redis is also provided with a timer cleaner for regularly cleaning up historical data: the timer cleaner regularly cleans up the historical data of the vehicle according to the set time.

[0013] The Redis is also provided with a variable monitor to monitor and clean up the global variables generated during the write cache or operation period: the variable monitor obtains the occupancy of the global variable memory at a specified frequency. When the occupancy ratio reaches a threshold, the variable monitor issues a cleanup instruction to clean up the historical data generated earlier in the global variable to reduce the occupancy ratio to the set value.

[0014] The summary calculator extracts the intermediate vehicle data and performs calculations based on the scheduled task framework:

[0015] The timed task framework includes iov-task-api, register and timed container, which divides the vehicle intermediate data into n tasks, including the following operations:

[0016] 1) iov-task-api provides a task-oriented interface, which includes the DefaultScheduleTask class and DefaultTask. The DefaultScheduleTask class includes lock and scheduling methods, and DefaultTask provides a business writing method for task execution. Tasks inherit the DefaultScheduleTask class, and then inherit DefaultTask. Tasks are released after meeting the unlocking conditions and enter the registrar through the interface for registration.

[0017] 2) The registrar includes task details and triggers. When registering, the task details name and group the tasks, and the trigger specifies the execution frequency of the tasks. After registration, the registrar notifies the timing container of the registration content of the task, and the timing container writes the task into the database according to the business writing method.

[0018] 3) The timing container also includes interfaces for starting, pausing, updating, deleting and displaying tasks; the display interface displays on the display interface by calling the display interface of the timing container, and performs corresponding operations by calling the task starting, pausing, updating and deleting interfaces.

[0019] Compared with the prior art, the present invention has the following beneficial technical effects:

[0020] The vehicle data real-time aggregation method provided by the present invention consumes the data source from the Kafka cluster to ensure the real-time performance of the data source; then the data results are summarized based on the Kafka unbounded flow window, and after the data source is obtained, grouping operations and aggregation are performed based on the real-time operation summary; the data summary can reach calculations in 20 dimensions, and the results are cached in redis storage respectively, so that the operation results can be continuously written to the redis cache, and redis is used as a cache database that can be written in an infinite loop, and then read and updated to the relational database by a scheduled task.

[0021] The real-time vehicle data aggregation method provided by the present invention adopts redis storage for the initial business variable objects across windows to ensure the correctness of the initial business variables after the program is restarted abnormally; after all calculation results are written to the redis cache, in order to ensure continuous data writing, the timed invalidation of the redis key is adopted; for monitoring the global variable state, a timer is used to obtain the memory occupancy at a specified frequency, and a timed task is used to clean up expired variables.

[0022] The vehicle data real-time aggregation method provided by the present invention can realize the aggregation and update of the latest data, and the total time of the entire operation and writing cache to persistence is controlled within a delay of less than 10 minutes. Compared with T+1 aggregation, the data is more real-time, the operation results are continuously updated, and the efficiency is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the process of the present invention;

[0024] Figure 2 Summarize the flow chart for scheduled tasks;

[0025] Figure 3 The schematic diagram of the unlocking process for releasing the task;

[0026] Figure 4 It is a schematic diagram of grouping and calculation results;

[0027] Figure 5 This is a schematic diagram of caching to the redis database;

[0028] Figure 6 A schematic diagram of continuous accumulation into the database. DETAILED DESCRIPTION

[0029] The present invention is further described in detail below in conjunction with the embodiments, which are intended to explain the present invention rather than to limit it.

[0030] See also Figure 1 The present invention provides a method for real-time aggregation of vehicle data, comprising the following operations:

[0031] 1) Real-time vehicle data is sent to the gateway through the vehicle terminal, and the gateway sends it to Kafka. The real-time vehicle data is saved as a binary unbounded stream;

[0032] 2) The data of the binary unbounded stream is then received by the parser, which parses it into plain text according to the specified business method. The business method is set in advance according to the processing needs of the business. For example, different business methods are used to parse different businesses such as mileage and fuel consumption.

[0033] 3) The grouping operator receives the plain text and groups it according to the data number and data time (Map); and calculates the grouped data in groups according to the business method (Reduce) to obtain the vehicle intermediate data including the data number, data time, timestamp (including start time, end time) and business content;

[0034] 4) Cache the intermediate vehicle data to Redis;

[0035] 5) The summary calculator periodically extracts the intermediate data of the vehicle from the Redis cache, performs calculations according to the business method, writes the calculation results to the database, and updates the vehicle business status.

[0036] Furthermore, due to the particularity of real-time computing, the following settings are made on Redis to ensure the accuracy, speed, and efficiency of the computing:

[0037] When the intermediate data of the vehicle is cached in Redis, the starting business status of the vehicle is saved in the Redis persistent area to prevent the loss of memory data (persistence within a certain period of time); when an exception occurs, the starting business status data of the vehicle will be loaded into the next computing task.

[0038] The Redis is also provided with a timer cleaner for cleaning up historical data at a fixed time: the timer cleaner cleans up the historical data of the vehicle at a fixed time according to the set time, for example, cleaning up the historical data of yesterday today.

[0039] The Redis is also provided with a variable monitor to monitor and clean up the global variables generated during the write cache or operation period: the variable monitor obtains the occupancy of the global variable memory at a specified frequency. When the occupancy ratio reaches a threshold, the variable monitor issues a cleanup instruction to clean up the historical data generated earlier in the global variable to reduce the occupancy ratio to the set value.

[0040] For further information, see Figure 2 The summary calculator extracts the intermediate vehicle data and performs calculations based on the scheduled task framework:

[0041] The timed task framework includes iov-task-api, register and timed container, which divides the vehicle intermediate data into n tasks, including the following operations:

[0042] 1) iov-task-api provides a task-oriented interface, which includes the DefaultScheduleTask class and DefaultTask. The DefaultScheduleTask class includes lock and scheduling methods, and DefaultTask provides a business writing method for task execution. Tasks inherit the DefaultScheduleTask class, and then inherit DefaultTask. Tasks are released after meeting the unlocking conditions and enter the registrar through the interface for registration.

[0043] 2) The registrar includes task details and triggers. When registering, the task details name and group the tasks, and the trigger specifies the execution frequency of the tasks. After registration, the registrar notifies the timing container of the registration content of the task, and the timing container writes the task into the database according to the business writing method.

[0044] 3) The timing container also includes interfaces for starting, pausing, updating, deleting and displaying tasks; the display interface displays on the display interface by calling the display interface of the timing container, and performs corresponding operations by calling the task starting, pausing, updating and deleting interfaces.

[0045] See also Figure 3 , the following takes task A as an example to illustrate the locking and unlocking process:

[0046] 1) When task A successfully sets lockkey through the setnx(lockkey,currenttime+timeout) command, the return value is true_1;

[0047] Set an expiration time for task lock A, execute the task of the task lock, complete it within the specified time and release it through the interface; when the task is blocked and the task lock times out, the key is forcibly released and unlocked through the interface

[0048] 2) When A cannot successfully enter false_1 through the setnx(lockkey,currenttime+timeout) command, since the lock timeout is set when the lock is set, when the current time is greater than the value of the lockkey stored in redis, it can be considered that the previous owner's right to use the lock has expired, and A can forcibly own the lock; the specific judgment process is as follows:

[0049] Task A uses get(lockkey) to obtain the value of lockkey stored in redis, that is, the relative time lockvalueA for acquiring the lock, lockvalueA! = null&¤ttime>lockvalue. Task A compares the current time with the time set for the lock. If the current time is greater than the time limit set for the lock, it can be further determined whether the lock can be acquired. Otherwise, it means that the lock is still occupied, and Task A cannot acquire the lock. It returns false_2, and the lock acquisition fails.

[0050] When the lock can be acquired and the return result is true_2, a new timeout is set through getSet, and the old value lockvalueB is returned for judgment. Because in a distributed environment, when entering here, another process may acquire the lock and modify the value. Only when the old value is consistent with the returned value can it be said that the lock has not been acquired by other processes in the middle;

[0051] lockvalueB==null||lockvalueA==lockvalueB, judgment: if lockvalueB is null, it means that the lock has been released, and the process can acquire the lock; if the old value is consistent with the returned lockvalueB, it means that the lock has not been acquired by other processes in the middle, and the lock can be acquired; otherwise, the lock cannot be acquired, and false_3 is returned at the end, and the lock acquisition fails;

[0052] Wait for the next unlock until the judgment is successful and pass through the interface in sequence.

[0053] Specific examples are given below.

[0054] A real-time vehicle data aggregation method includes the following operations:

[0055] 1. The data source of Kafka binary unbounded stream is as follows:

[0056] Data number Data time Pulse speed Engine speed … 7EA2689 268F901 8D65A 7BB5E … 7EA2689 2698084 EOAB6 0 … 7EB80AC 8738428 D238EB 0 … 7EB80AC 3745393 0 345433 … …

[0057] 2. Analysis

[0058] The above table data is parsed into decimal plain text according to business rules, in the following format:

[0059]

[0060]

[0061] 3. Group calculation

[0062] The grouping and calculation results are as follows Figure 4As shown, MAP: the table data above is grouped according to the first column; REDUCE: the grouped data is aggregated according to business rules;

[0063] 4. Computation of intermediate results and cache

[0064] like Figure 5 As shown, the results of REDUCE are cached in the redis database;

[0065] 5. Database summary update

[0066] Scheduled tasks aggregate data into the database (Oracle)

[0067] Figure 5 The data list shown needs to be continuously accumulated into the Oracle result table. After the accumulation, the redis cache will be deleted. The result after the accumulation is as follows Figure 6 shown.

[0068] More specifically, the following is an example of fuel consumption per 100 kilometers:

[0069] In the T+1 calculation method, only the total mileage and fuel consumption of yesterday can be calculated on the same day, and the fuel consumption per 100 kilometers of each vehicle can be calculated on this basis;

[0070] By adopting the real-time aggregation of the present invention, the total vehicle mileage and total fuel consumption 10 minutes ago can be obtained, and the fuel consumption value per 100 kilometers 10 minutes ago can be directly calculated on this basis; and it can be summarized again in the next cycle.

[0071] The following is an explanation based on the running time:

[0072] In units of 10 minutes, data is obtained from Kafka, parsed, grouped, cached, and summarized at regular intervals; the data is summarized again in the next cycle; the results of each calculation are accumulated, and the cumulative results for the day are generated in the last calculation.

[0073] Different businesses have different aggregation frequencies and methods (depending on the business type). The calculations in other dimensions are similar to the above comparison, including: gas consumption per 100 kilometers, idling time, fatigue driving time, number of sudden acceleration and deceleration, refueling, abnormal fuel volume, number of high throttle times, cold car driving time, number of times the accelerator is pressed when stopping, number of times the accelerator is pressed hard, immediate engine shutdown when stopping, speeding, etc.

[0074] The vehicle data real-time aggregation method provided by the present invention can realize the aggregation and update of the latest data, and the total time of the entire operation and writing cache to persistence is controlled within a delay of less than 10 minutes. Compared with T+1 aggregation, the data is more real-time, the operation results are continuously updated, and the efficiency is higher.

[0075] The above embodiments are preferred examples for implementing the present invention, and the present invention is not limited to the above embodiments. Any non-essential additions and substitutions made by those skilled in the art based on the technical features of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A real-time vehicle data aggregation method, characterized in that: The following operations are included: 1) Real-time vehicle data is sent to the gateway through the vehicle terminal, and the gateway sends it to Kafka. The real-time vehicle data is saved as a binary unbounded stream; 2) The data of the binary unbounded stream is then received by the parser, which parses it into plain text according to the prescribed business method; 3) The grouping operator receives the plain text and groups it according to the data number and data time; The grouped data are calculated in groups according to the business method to obtain the vehicle intermediate data including data number, data time, timestamp and business content; 4) Cache the intermediate vehicle data to Redis; 5) The summary calculator periodically extracts the intermediate data of the vehicle from the Redis cache, performs calculations according to the business method, writes the calculation results into the database, and updates the business status of the vehicle; When the intermediate data of the vehicle is cached in Redis, the starting business status of the vehicle is saved in the Redis persistent area to prevent the loss of memory data; when an exception occurs, the starting business status data of the vehicle will be loaded into the next computing task; The Redis is also provided with a timer cleaner for regularly cleaning up historical data: the timer cleaner regularly cleans up the historical data of the vehicle according to the set time.

2. The method for real-time aggregation of vehicle data according to claim 1, characterized in that: The Redis is also provided with a variable monitor to monitor and clean up the global variables generated during the write cache or operation period: the variable monitor obtains the occupancy of the global variable memory at a specified frequency. When the occupancy ratio reaches a threshold, the variable monitor issues a cleanup instruction to clean up the historical data generated earlier in the global variable to reduce the occupancy ratio to the set value.

3. The method for real-time aggregation of vehicle data according to claim 1, characterized in that: The summary calculator extracts the intermediate vehicle data and performs calculations based on the scheduled task framework: The timed task framework includes iov-task-api, register and timed container, which divides the vehicle intermediate data into n tasks, including the following operations: 1) iov-task-api provides a task-oriented interface, which includes the DefaultScheduleTask class and DefaultTask. The DefaultScheduleTask class includes lock and scheduling methods, and DefaultTask provides a business writing method for task execution. Tasks inherit the DefaultScheduleTask class, and then inherit DefaultTask. Tasks are released after meeting the unlocking conditions and enter the registrar through the interface for registration. 2) The registrar includes task details and triggers. When registering, the task details name and group the tasks, and the trigger specifies the execution frequency of the tasks. After registration, the registrar notifies the timing container of the registration content of the task, and the timing container writes the task into the database according to the business writing method. 3) The timing container also includes interfaces for starting, pausing, updating, deleting and displaying tasks; the display interface displays on the display interface by calling the display interface of the timing container, and performs corresponding operations by calling the task starting, pausing, updating and deleting interfaces.

Images