Vehicle driving mileage batch processing method and system based on open source big data component

Abstract

The invention discloses a vehicle driving mileage batch processing method and system based on an open source big data component, and relates to the technical field of big data, and the technical scheme is characterized in that GPS positioning data in a database is synchronized to Hive in batches; the method comprises the following steps: compressing GPS positioning data into a gplist field of one row by using a function of converting multiple rows into one column and built in Hive; compiling the UDF function into a jar package and deploying the jar package into Hive; and utilizing a UDF function to calculate the vehicle driving mileages of all vehicles in the first temporary table in the previous time period in batches through Hive SQL, and storing the calculation result in a second temporary table. According to the method, line-to-column and user-defined UDF functions of Hive are adopted, the travelled distance of a large number of vehicles can be calculated in a short time through a distributed cluster, and the calculation result of the travelled distance of million-level vehicles in the previous day can be obtained in minute-level time.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and more specifically, to a vehicle mileage batch processing method and system based on open source big data components. Background Art

[0002] In the post-loan management of commercial auto loans in the financial industry, it is often necessary to collect GPS positioning data (or other satellite positioning data) of vehicles, and regularly monitor the location, trajectory, mileage and other information of vehicles based on these GPS data, so as to timely discover possible auto loan risks and abnormal situations. Among them, regular statistics and monitoring of vehicle mileage is a very important risk control method. However, since the number of vehicles in the auto loan business of financial institutions is often very large (up to millions), and for the need for accurate monitoring, GPS data is usually collected every few seconds (calculated at every 10 seconds, a car can have up to 8,640 pieces of GPS data per day), which leads to the need to regularly count (for example, once a day) the detailed data involved in the daily mileage of all vehicles is very large (up to tens of billions), and the calculation is very time-consuming and resource-consuming.

[0003] The existing technologies related to vehicle mileage calculation are almost all focused on how to improve the accuracy of single-vehicle mileage calculation, rather than solving how to efficiently and quickly calculate vehicle mileage in batches in the context of massive data. One method that is easier to think of is to store each piece of positioning data of these vehicles in a common relational database such as MySQL and PostgreSQL, and write a stand-alone program in a common programming language such as JAVA and Python. Each time, all the positioning data of a certain vehicle on a certain day is read from the database and sorted in chronological order. The distance between two adjacent time points is calculated and accumulated to obtain the mileage of the vehicle on that day. Then the mileage of the next vehicle is calculated in turn until the mileage of all vehicles is traversed and calculated. However, this method is very inefficient. It usually takes several days to obtain the calculation results for millions of vehicles, which is difficult to meet the requirements of risk control.

[0004] Therefore, how to research and design a vehicle mileage batch processing method and system based on open source big data components that can overcome the above-mentioned defects is a problem that we urgently need to solve. Summary of the invention

[0005] In order to address the deficiencies in the prior art, the purpose of the present invention is to provide a vehicle mileage batch processing method and system based on open source big data components, which adopts Hive's row-to-column and custom UDF functions, and can use distributed clusters to calculate the mileage of massive vehicles in a relatively short time. The mileage of millions of vehicles in the previous day can be calculated within minutes.

[0006] The above technical objectives of the present invention are achieved through the following technical solutions:

[0007] In the first aspect, a vehicle mileage batch processing method based on an open source big data component is provided, comprising the following steps:

[0008] Batch synchronize the GPS positioning data of all vehicles in the previous time period in the database to Hive to obtain the gps table;

[0009] Use the built-in multi-row to one-column function in Hive to compress all GPS positioning data of each vehicle in the previous time period in the gps table into a row of gps_list field and write it to the first temporary table;

[0010] Use JAVA language to develop a UDF function to calculate the mileage of a single vehicle based on the gps_list field in the temporary table, compile the UDF function into a jar package and deploy it to Hive;

[0011] The UDF function is used to batch calculate the vehicle mileage of all vehicles in the first temporary table in the previous time period through Hive SQL, and the calculation results are stored in the second temporary table.

[0012] Furthermore, the fields in the gps table include vehicle ID, longitude of current vehicle position, latitude of current vehicle position and current timestamp.

[0013] Furthermore, the function for converting multiple rows into one column is a collect_set function or a collect_list function.

[0014] Furthermore, the input parameter of the UDF function is the gps_list field, the return value is the vehicle mileage, and the calculation logic includes:

[0015] Split and sort the compressed and stored single gps_list field;

[0016] Use the spherical distance formula to calculate the distance between two adjacent points;

[0017] After excluding the distance beyond the normal range, the distance is accumulated to obtain the vehicle mileage.

[0018] Furthermore, the process of splitting and sorting the compressed and stored single gps_list field is specifically as follows:

[0019] Split a gps_list field value into an array based on the first separator, where each element in the array represents the GPS location of the vehicle at a certain moment;

[0020] Split each element into an object containing three fields according to the second separator: longitude, latitude and timestamp;

[0021] Sort all objects in the array from early to late according to the timestamp to get the sorting result.

[0022] Furthermore, the expression for calculating the distance between two adjacent points using the spherical distance formula is specifically:

[0023] L=6371×arccos[cosA2×cosB2×cos(A1-B1)+sinA2×sinB2];

[0024] Among them, L represents the distance between point A and point B; A1 represents the longitude of point A; A2 represents the latitude of point A; B1 represents the longitude of point B; B2 represents the latitude of point B.

[0025] Furthermore, the process of excluding distances beyond the normal range is specifically as follows:

[0026] If the distance between a point and the two adjacent points before and after it exceeds the normal range, the element corresponding to the point will be removed from the array;

[0027] If the distance between a point and the previous point exceeds the normal range, and the distance between the point and the next point does not exceed the normal range, the distance between the point and the previous point will not be included in the total mileage calculation;

[0028] If the distance between a point and the next point exceeds the normal range, and the distance between the point and the previous point does not exceed the normal range, the distance between the point and the next point will not be included in the total mileage calculation.

[0029] In a second aspect, a vehicle mileage batch processing system based on an open source big data component is provided, the system is used to implement the vehicle mileage batch processing method based on an open source big data component as described in any one of the first aspects, including:

[0030] The data synchronization module is used to batch synchronize the GPS positioning data of all vehicles in the previous time period in the database to Hive to obtain the gps table;

[0031] The data compression module is used to compress all GPS positioning data of each vehicle in the gps table in the previous time period into a row of the gps_list field using the built-in multi-row to one-column function in Hive, and write it to the first temporary table;

[0032] Function deployment module, used to develop a UDF function using JAVA language to calculate the mileage of a single vehicle based on the gps_list field in the temporary table, compile the UDF function into a jar package and deploy it to Hive;

[0033] The batch calculation module is used to use the UDF function to batch calculate the vehicle mileage of all vehicles in the first temporary table in the previous time period through Hive SQL, and store the calculation results in the second temporary table.

[0034] In a third aspect, a computer terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the vehicle mileage batch processing method based on open source big data components as described in any one of the first aspects is implemented.

[0035] In a fourth aspect, a computer-readable medium is provided, on which a computer program is stored, and the computer program is executed by a processor to implement the vehicle mileage batch processing method based on open source big data components as described in any one of the first aspects.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The vehicle mileage batch processing method based on open source big data components provided by the present invention adopts the row-to-column and custom UDF functions of Hive. Since Hive is a distributed parallel computing framework, it has efficient batch computing capabilities in massive data scenarios and has very high throughput. Distributed clusters can be used to calculate the mileage of massive vehicles in a short time, and the mileage of millions of vehicles the day before can be calculated within minutes. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0039] Figure 1 is a flow chart of Embodiment 1 of the present invention;

[0040] Figure 2 It is a system block diagram in Example 2 of the present invention. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0042] Apache Hive: A data warehouse analysis system built on Hadoop. It provides a rich set of SQL query methods to analyze data stored in the Hadoop distributed file system: it can map structured data files to a database table and provide complete SQL query functions; it can convert SQL statements into MapReduce tasks and analyze the required content through its own SQL queries. This set of SQL is referred to as Hive SQL, which allows users who are not familiar with MapReduce to easily use SQL language to query, summarize and analyze data.

[0043] UDF: User-defined function, it is a powerful feature that allows users to extend Hive SQL, and some other types of databases also have similar features. Users can write their own UDFs in Java. Once the UDF is added to the user session (interactively or through a script), they will be used like built-in functions and even provide online help. Hive has multiple types of UDFs, each of which performs a specific "class" of transformation processes on the input data. In ETL processing, a process may contain multiple processing steps. The Hive language has multiple ways to pass the input of the previous step to the next step through a pipeline, and then produce multiple outputs in a query. Users can also write custom functions for some specific processes. Without this function, a process may need to include a MapReduce step or need to transfer data to another system to implement these changes. Therefore, Hive provides UDFs, which are executed in the same task process generated by the Hive query, so they can be executed efficiently.

[0044] Here is a description of the vehicle GPS data format used as raw data input. Every few seconds, each vehicle sends a piece of GPS positioning data of the current vehicle to the data collection center, which in turn sends the data to the financial institution. The data contains at least four fields: vehicle ID (which can uniquely identify a vehicle, which can be understood as the license plate number, hereinafter referred to as vehicle_id), longitude of the current vehicle position (hereinafter referred to as vehicle_longitude), latitude of the current vehicle position (hereinafter referred to as vehicle_latitude), and current timestamp (hereinafter referred to as timestamp). As for other fields, they are not related to mileage calculation, so they are omitted here. These data will be stored in the database, and each row in the database corresponds to the above piece of GPS positioning data.

[0045] Example 1: A vehicle mileage batch processing method based on open source big data components, such as Figure 1 As shown, the following steps are included:

[0046] Step 1: synchronize the GPS positioning data of all vehicles in the database in the previous time period to Hive in batches to obtain the gps table;

[0047] Step 2: Use the built-in multi-row to one-column function in Hive to compress all GPS positioning data of each vehicle in the previous time period in the gps table into a row of gps_list field and write it to the first temporary table;

[0048] Step 3: Use JAVA to develop a UDF function that calculates the mileage of a single vehicle based on the gps_list field in the temporary table, compile the UDF function into a jar package, and deploy it to Hive;

[0049] Step 4: Use the UDF function to batch calculate the vehicle mileage of all vehicles in the first temporary table in the previous time period through Hive SQL, and store the calculation results in the second temporary table.

[0050] Here we take the calculation of the mileage of all vehicles once a day as an example. Calculation once a day (T+1) is the most typical application scenario. As for other time periods (weekly, hourly, etc.), the principle is similar.

[0051] Step 1: Batch synchronize the GPS positioning data of all vehicles in the database from the previous day to Hive. It can be synchronized once a day, or once every hour or other time periods. The Hive table name is assumed to be gps. The gps table contains 4 main fields: vehicle ID (which can uniquely identify a vehicle, which can be understood as the license plate number, hereinafter referred to as vehicle_id), the longitude of the current vehicle position (hereinafter referred to as vehicle_longitude), the latitude of the current vehicle position (hereinafter referred to as vehicle_latitude), and the current timestamp (hereinafter referred to as timestamp). Other fields are not related to the mileage calculation recorded in the present invention, so they are not synchronized to the gps table.

[0052] Step 2: Use the built-in multi-row-to-one-column function collect_set or collect_list in Hive to compress all GPS positioning data of each vehicle on the previous day into one row of data, and then write it to another temporary table (assuming it is named gps_collection). The specific method is: execute the following SQL statement in Hive:

[0053] CREATE TABLE gps_collection AS SELECT

[0054] vehicle_id,concat_ws(',',collect_set(concat(vehicle_longitude,'+',vehicle_latitude,'+',timestamp))AS gps_list FROM gps GROUP BY vehicle_id;

[0055] The above creates a new table called gps_collection, which contains the latitude, longitude and timestamp information of each vehicle from the gps table, grouped by vehicle_id, and merged into a list.

[0056] The specific steps are as follows:

[0057] 1. Select the vehicle_id field from the gps table.

[0058] 2. Use the collect_set function to collect all latitude, longitude and timestamp information corresponding to each vehicle_id.

[0059] 3. Connect the vehicle_longitude, vehicle_latitude and timestamp of each vehicle with +, in the format of longitude + latitude + timestamp.

[0060] 4. Use the concat_ws function to separate all the latitude, longitude and timestamp information collected for each vehicle with commas to form the gps_list column.

[0061] 5. Group by vehicle_id and generate a gps_list column for each group.

[0062] Ultimately, the gps_collection table will contain two columns: vehicle_id and gps_list. The gps_list column contains all the latitude, longitude and timestamp information for each vehicle, separated by commas.

[0063] ',' and '+' are field separators, and other separators can also be used. In the gps_list field of the generated gps_collection table, each row stores all the GPS positioning data of a vehicle on the previous day.

[0064] After step 2, the number of rows in the gps_collection table is greatly reduced compared to the gps table. For example, if there are 1 million vehicles and each vehicle has 10,000 GPS positioning data, the gps table will have 10 billion rows, while the gps_collection table will have only 1 million rows.

[0065] Step 3: Use JAVA language to develop a UDF (Hive user-defined function) that can calculate the mileage of a single vehicle based on the gps_list field. Assume that the UDF is named calculate_mileage, the full function signature is double calculate_mileage(string gps_list), the function has only one input parameter, namely gps_list, and the function return value is the mileage of a certain vehicle. Based on the idea of ​​divide and conquer, this UDF function handles the mileage calculation of a single vehicle, providing a basis for the next step of using Hive to efficiently batch calculate the mileage calculation of all vehicles.

[0066] The calculation logic of the UDF function is divided into the following steps:

[0067] 3.1) Split a gps_list field value into an array (or list) based on the "," separator. Each element in the array represents the GPS location of the vehicle at a certain moment.

[0068] 3.2) Split each element value into an object containing three fields based on the "+" separator: longitude, latitude, and timestamp.

[0069] 3.3) Sort all objects in the array from earliest to latest according to their timestamps.

[0070] 3.4) According to the sorting results, calculate the distance between two adjacent objects (representing the position of a vehicle at the previous moment and the position at the next moment). Since the shape of the earth is very close to a sphere, with an average radius of 6371 kilometers, and the distance traveled by a vehicle in a few seconds is not too long (generally less than 1 kilometer), this paper uses the spherical distance formula, which will not produce a large error.

[0071] Assuming that the longitude of point A is A1 and the latitude is A2, and the longitude of point B is B1 and the latitude is B2, the distance L (in kilometers) between points AB is equal to: L = 6371×arccos[cosA2×cosB2×cos(A1-B1)+sinA2×sinB2].

[0072] 3.5) Perform an abnormality check on the distance between each two adjacent points calculated in the previous step. If the distance is found to be too large and exceeds the normal range (which can be judged by the speed limit of 200 kilometers per hour, that is, 55.6 meters per second), it is considered abnormal. If the distance between a point and the two adjacent points before and after it exceeds the normal range, the element corresponding to the point is removed from the array; if the distance between a point and the previous point exceeds the normal range, but the distance between the point and the next point does not exceed the normal range, the distance between the point and the previous point is not included in the total mileage calculation; if the distance between a point and the next point exceeds the normal range, but the distance between the point and the previous point does not exceed the normal range, the distance between the point and the next point is not included in the total mileage calculation.

[0073] 3.6) Add up the distances between each two adjacent points to get the total mileage of a vehicle the previous day. Suppose a vehicle reports n GPS positioning data in a day, and the position corresponding to the i-th positioning data is P i Point, P i The longitude is P i 1. Latitude is P i 2, then the mileage of the car on a certain day is:

[0074] It can be seen that the above calculation logic uses the ideas of subdivision, summation and approximation in calculus.

[0075] Through the above 6 steps, the calculation of the mileage of a single vehicle on a certain day is realized. Compile the developed calculate_mileage function into a jar package and deploy it to Hive. The development and deployment of the calculate_mileage function is a one-time work, and this function can be reused each time the mileage is calculated later.

[0076] Step 4: Use the calculate_mileage function developed in step 3 to batch calculate the mileage of all vehicles in the gps_collection table on the previous day through Hive SQL, and store the calculation results in another table (assuming it is named gps_mileage). The SQL statement is written as follows:

[0077] CREATE TABLE gps_mileage AS SELECT vehicle_id,calculate_mileage(gps_list)AS vehicle_mileage FROM gps_collection;

[0078] The above creates a new table called gps_mileage that contains the vehicle_id selected from the gps_collection table and the calculated mileage information.

[0079] The specific steps are as follows:

[0080] 1. Select the vehicle_id field from the gps_collection table.

[0081] 2. For each vehicle_id, use the calculate_mileage function to calculate the mileage in its corresponding gps_list field.

[0082] 3. Name the calculated mileage column vehicle_mileage.

[0083] Ultimately, the gps_mileage table will contain two columns: vehicle_id and vehicle_mileage, where the vehicle_mileage column contains the mileage of each vehicle based on its GPS listing.

[0084] Each row in the gps_mileage table represents the total mileage of a vehicle on the previous day, and the vehicle_mileage field stores the mileage value of the vehicle corresponding to the value of the vehicle_id field.

[0085] Embodiment 2: A vehicle mileage batch processing system based on an open source big data component, the system is used to implement the vehicle mileage batch processing method based on an open source big data component as described in Embodiment 1, such as Figure 2 As shown, it includes a data synchronization module, a data compression module, a function deployment module and a batch computing module.

[0086] Among them, the data synchronization module is used to synchronize the GPS positioning data of all vehicles in the database in the previous time period to Hive in batches to obtain the gps table; the data compression module is used to use the built-in multi-row to one-column function in Hive to compress all GPS positioning data of each vehicle in the gps table in the previous time period into a row of gps_list field, and write it to the first temporary table; the function deployment module is used to use JAVA language to develop a UDF function for calculating the mileage of a single vehicle based on the gps_list field in the temporary table, compile the UDF function into a jar package and deploy it to Hive; the batch calculation module is used to use the UDF function to batch calculate the vehicle mileage of all vehicles in the first temporary table in the previous time period through Hive SQL, and store the calculation results in the second temporary table.

[0087] The present invention also records a computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the vehicle mileage batch processing method based on open source big data components as described in Example 1 is implemented.

[0088] The present invention also records a computer-readable medium on which a computer program is stored. The computer program is executed by a processor to implement the vehicle mileage batch processing method based on open source big data components as described in Example 1.

[0089] Working principle: The present invention adopts Hive's row-to-column and custom UDF functions. Since Hive is a distributed parallel computing framework, it has efficient batch computing capabilities in massive data scenarios and a very high throughput. Distributed clusters can be used to calculate the mileage of massive vehicles in a short time. The mileage of millions of vehicles the day before can be calculated within minutes.

[0090] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0091] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0092] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0094] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A vehicle mileage batch processing method based on open source big data components, characterized by: The following steps are involved: Batch synchronize the GPS positioning data of all vehicles in the previous time period in the database to Hive to obtain the gps table; Use the built-in multi-row to one-column function in Hive to compress all GPS positioning data of each vehicle in the previous time period in the gps table into a row of gps_list field and write it to the first temporary table; Use JAVA language to develop a UDF function to calculate the mileage of a single vehicle based on the gps_list field in the temporary table, compile the UDF function into a jar package and deploy it to Hive; The UDF function is used to batch calculate the vehicle mileage of all vehicles in the first temporary table in the previous time period through Hive SQL, and the calculation results are stored in the second temporary table.

2. The vehicle mileage batch processing method based on open source big data components according to claim 1 is characterized in that: The fields in the gps table include vehicle ID, longitude of current vehicle position, latitude of current vehicle position and current timestamp.

3. The vehicle mileage batch processing method based on open source big data components according to claim 1 is characterized in that: The function for converting multiple rows into one column is a collect_set function or a collect_list function.

4. The vehicle mileage batch processing method based on open source big data components according to claim 1 is characterized in that: The input parameter of the UDF function is the gps_list field, and the return value is the vehicle mileage. The calculation logic includes: Split and sort the compressed and stored single gps_list field; Use the spherical distance formula to calculate the distance between two adjacent points; After excluding the distance beyond the normal range, the distance is accumulated to obtain the vehicle mileage.

5. The vehicle mileage batch processing method based on open source big data components according to claim 4 is characterized in that: The process of splitting and sorting the compressed and stored single gps_list field is specifically as follows: Split a gps_list field value into an array based on the first separator, where each element in the array represents the GPS location of the vehicle at a certain moment; Split each element into an object containing three fields according to the second separator: longitude, latitude and timestamp; Sort all objects in the array from early to late according to the timestamp to get the sorting result.

6. The vehicle mileage batch processing method based on open source big data components according to claim 4 is characterized in that: The expression for calculating the distance between two adjacent points using the spherical distance formula is specifically: L=6371×arccos[cosA2×cosB2×cos(A1-B1)+sinA2×sinB2]; Among them, L represents the distance between point A and point B; A1 represents the longitude of point A; A2 represents the latitude of point A; B1 represents the longitude of point B; B2 represents the latitude of point B.

7. The vehicle mileage batch processing method based on open source big data components according to claim 4 is characterized in that: The process of excluding distances beyond the normal range is specifically as follows: If the distance between a point and the two adjacent points before and after it exceeds the normal range, the element corresponding to the point will be removed from the array; If the distance between a point and the previous point exceeds the normal range, and the distance between the point and the next point does not exceed the normal range, the distance between the point and the previous point will not be included in the total mileage calculation; If the distance between a point and the next point exceeds the normal range, and the distance between the point and the previous point does not exceed the normal range, the distance between the point and the next point will not be included in the total mileage calculation.

8. The vehicle mileage batch processing system based on open source big data components is characterized by: The system is used to implement the vehicle mileage batch processing method based on open source big data components as described in any one of claims 1 to 7, including: The data synchronization module is used to batch synchronize the GPS positioning data of all vehicles in the previous time period in the database to Hive to obtain the gps table; The data compression module is used to compress all GPS positioning data of each vehicle in the gps table in the previous time period into a row of the gps_list field using the built-in multi-row to one-column function in Hive, and write it to the first temporary table; Function deployment module, used to develop a UDF function using JAVA language to calculate the mileage of a single vehicle based on the gps_list field in the temporary table, compile the UDF function into a jar package and deploy it to Hive; The batch calculation module is used to use the UDF function to batch calculate the vehicle mileage of all vehicles in the first temporary table in the previous time period through Hive SQL, and store the calculation results in the second temporary table.

9. A computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the vehicle mileage batch processing method based on open source big data components as described in any one of claims 1-7.

10. A computer readable medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the vehicle mileage batch processing method based on open source big data components as described in any one of claims 1 to 7.

Images