Internet of Vehicles data analysis platform based on big data architecture

Abstract

The invention discloses an Internet of Vehicles data analysis platform based on a big data architecture. The platform comprises a data receiving module, a data analysis task management module and a data pushing module, the data receiving module receives the collected data reported by a vehicle-mounted terminal, judges and processes the collected data according to a certain data preprocessing logic, and writes the data meeting a certain condition into message middleware Kafka; the data analysis task management module submits a data analysis task to a Spark distributed computing engine, performsunified monitoring and management in the execution process of the data analysis task, and writes monitoring indexes and a final analysis result in a database Clickhouse in the operation period of thedata analysis task; the data pushing module receives a request that a user subscribes a certain data analysis task to output a result based on a Websocket protocol; and the problems that a traditional data analysis platform cannot be connected with an online big data scene environment, and the model training analysis process is not standard are solved.

Description

technical field [0001] The invention relates to a car networking data analysis platform based on a big data architecture, and belongs to the technical field of big data car networking data analysis. Background technique [0002] With the continuous implementation and improvement of industrial policies and the maturity and popularization of vehicle-mounted smart terminal technology, more and more companies have completed the reception and storage of vehicle data; however, in the face of massive data reported by terminals, most of the current applications are Do some processing on current real-time data, or do some simple statistical analysis on historical data. However, in stark contrast to this is the high cost of platform construction and maintenance and storage costs; Information is reported once every 10 seconds. If each piece of information is 1KB (after compression), the storage requirement for one year is about 3PB, and the storage cost is nearly one million yuan. [...

AI Technical Summary

Problems solved by technology

[0004] The present invention proposes a car networking data analysis platform based on a big data architecture, and its purpose is to solve the problems that the traditional data analysis platform cannot be connected with the online environment, the model training and analysis process is not standardized, and the stand-alone processing performance is low.

Images