Method and system for processing flow batch integrated data in vehicle

Abstract

The invention discloses a method and a system for processing flow batch integrated data in a vehicle, and belongs to the technical field of data processing. The Kafka component obtains real-time data reported by the vehicle; the Flink engine reads the real-time data from the Kafka component, and writes the real-time data into a data service layer of the Paimon component in a streaming manner according to a first path; the data synchronization tool uses an Flink engine to obtain offline data provided by an external data source; the Flink engine writes the offline data into a data service layer of the Paimon component in batches according to a second path; and the Flink engine writes the real-time data and the offline data in the data service layer into the index library. Stream batch integrated data processing is realized through the Flink engine, the operation and maintenance cost, the learning cost and the development cost are reduced, the processing delay is reduced, unified storage and access of data are realized through the Paimon component, the data consistency is ensured, and the persistence capability is provided for the data.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and system for processing integrated flow and batch data in a vehicle. Background Art

[0002] In big data processing, Lambda architecture is a common architecture pattern. Lambda architecture includes batch processing layer, speed processing layer and service layer. The batch processing layer is an offline computing link, which is used to process historical data in batches to obtain offline views; the speed processing layer is a real-time computing link, which is used to process real-time data streams to obtain real-time views; the service layer is used to merge the views of the batch processing layer and the speed processing layer to obtain a unified data view.

[0003] like Figure 1 As shown in the figure, the offline computing link adopts the traditional distributed system infrastructure Hadoop. After the external offline data is connected to the data warehouse, the Hadoop Distributed File System (HDFS) is used as storage. The Spark engine or the data warehouse tool Hive processes the offline data and synchronizes the final offline data to the relational database (DataBase, DB) or database Doris through the data service (DataWarehouse Service, DWS) layer; the real-time computing link uses the stream processing platform Kafka as the middle layer to store data. The Flink engine processes the real-time data. After passing through the operational data storage (Operational Data Store, ODS) layer, the data detail (DataWarehouse Detail, DWD) layer and the DWS layer, the final data is written into the relational DB or key-value (KV) database for use by the dashboard or data service. The offline link and the real-time link adopt different technical architectures to process large quantities of historical data and real-time data with high real-time requirements respectively.

[0004] However, the offline computing link and the real-time computing link are two systems, and two sets of codes need to be developed, which not only increases the operation and maintenance costs, but also increases the learning and development costs. The computing engines of the offline computing link and the real-time computing link are different, making it difficult to ensure data consistency; and the data storage formats are different, which increases the complexity of data management. In addition, the traditional Lambda architecture relies on the Hadoop cluster, which monopolizes resources and wastes computing resources during low data peaks. Summary of the invention

[0005] The present application provides a method and system for processing integrated stream and batch data in a vehicle, which is used to solve the problems of high operation and maintenance costs, learning costs and development costs, difficulty in ensuring data consistency, different data storage formats, and Hadoop cluster monopolizing resources and wasting computing resources during low data peaks when using Lambda architecture to process integrated stream and batch data. The technical solution is as follows:

[0006] According to a first aspect of the present application, a method for processing flow-batch integrated data in a vehicle is provided, the method comprising:

[0007] The Kafka component obtains real-time data reported by vehicles;

[0008] The Flink engine reads the real-time data from the Kafka component, and writes the real-time data into the data service layer of the Paimon component in a streaming manner according to the first path;

[0009] The data synchronization tool uses the Flink engine to obtain offline data provided by the external data source;

[0010] The Flink engine writes the offline data into the data service layer of the Paimon component in batches according to the second path;

[0011] The Flink engine writes the real-time data and offline data in the data service layer into the indicator library.

[0012] In a possible implementation, writing the real-time data into the data service layer of the Paimon component in a streaming manner according to the first path includes:

[0013] For the first data layer node in the first path, the Flink engine writes the real-time data into the first data layer node; the Paimon component generates incremental data and data snapshots according to the real-time data;

[0014] For the i-th data layer node in the first path, i≥2, the Flink engine reads the incremental data and data snapshot of the real-time data from the i-1-th data layer node, performs corresponding processing on the real-time data according to the incremental data and the data snapshot, and writes the processed real-time data to the i-th data layer node; the Paimon component generates incremental data and data snapshots based on the processed real-time data.

[0015] In a possible implementation, the Flink engine writes the real-time data into the first data layer node, including:

[0016] If the real-time data is data in Protobuf format stored in a key-value pair manner, the Flink engine converts the Value in Protobuf format into a Value in Json format;

[0017] The Flink engine extracts information from the key and adds it to the format-converted Value.

[0018] The Flink engine writes the supplemented Value into the first data layer node as real-time data.

[0019] In a possible implementation, the Flink engine writes the real-time data into the first data layer node, including:

[0020] The Flink engine reads the database information and data table information corresponding to the real-time data in the Piamon component;

[0021] The Flink engine detects whether there is a database corresponding to the database information in the first data layer node;

[0022] If there is a database corresponding to the database information, the Flink engine detects whether there is a data table corresponding to the data table information in the database;

[0023] If a data table corresponding to the data table information exists, the Flink engine writes the real-time data into the data table.

[0024] In a possible implementation, the Flink engine writes the offline data in batches into the data service layer of the Paimon component according to the second path, including:

[0025] For the first data layer node in the second path, the Flink engine writes the offline data into the first data layer node; the Paimon component generates a data snapshot according to the offline data;

[0026] For the i-th data layer node in the second path, i≥2, the Flink engine reads the data snapshot of the offline data from the i-1-th data layer node, performs corresponding processing on the offline data according to the data snapshot, and writes the processed offline data to the i-th data layer node; the Paimon component generates a data snapshot based on the processed offline data.

[0027] In a possible implementation, the method further includes: when initializing the first data layer node, the Flink engine creates a database and a data table corresponding to the offline data in the Piamon component;

[0028] The Flink engine writes the offline data into the first data layer node, including: the Flink engine writes the offline data into a data table of the database.

[0029] In a possible implementation, the Kafka component obtains the real-time data reported by the vehicle, including: the Kafka component receives the real-time data reported by the vehicle based on the message queue telemetry transmission MQTT protocol; the producer in the Kafka component sends the real-time data to the agent according to the topic, and the agent in the Kafka component stores the real-time data according to the topic;

[0030] The Flink engine reads the real-time data from the Kafka component, including: the Flink engine, as a consumer, obtains the real-time data of the subscribed topic from the Kafka component.

[0031] In a possible implementation, the method further includes:

[0032] The Superset component reads the real-time data and the offline data from the indicator library, and generates a query report or a data analysis report for the real-time data and the offline data.

[0033] In a possible implementation, the method further includes:

[0034] Deploy the Flink engine on the Kubernetes cluster;

[0035] Install the Paimon component;

[0036] Add the Jar package of the Paimon component to the directory of the Flink engine so that the Flink engine can access the Paimon component.

[0037] According to a second aspect of the present application, a system for processing stream-batch integrated data in a vehicle is provided, wherein the system for processing stream-batch integrated data includes a Kafka component, a Flink engine, and a data synchronization tool;

[0038] The Kafka component is used to obtain real-time data reported by vehicles;

[0039] The Flink engine is used to read the real-time data from the Kafka component and write the real-time data into the data service layer of the Paimon component in a streaming manner according to the first path;

[0040] A data synchronization tool, used to use the Flink engine to obtain offline data provided by an external data source;

[0041] The Flink engine is further used to write the offline data into the data service layer of the Paimon component in batches according to the second path;

[0042] The Flink engine is also used to write the real-time data and offline data in the data service layer into the database.

[0043] According to a third aspect of the present application, a computer-readable storage medium is provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the method for processing integrated flow and batch data in a vehicle as described above.

[0044] According to a fourth aspect of the present application, a computer device is provided, the computer device comprising the above-mentioned system for processing flow-batch integrated data in a vehicle.

[0045] The beneficial effects of the technical solution provided by this application include at least:

[0046] The Flink engine writes real-time data into the data service layer of the Paimon component in a streaming manner along the first path; the Flink engine can write offline data into the data service layer of the Paimon component in batches along the second path; then, the Flink engine writes the real-time data and offline data in the data service layer into the indicator library. In this way, the Flink engine can be used to implement integrated streaming and batch data processing, which not only reduces operation and maintenance costs, learning costs, and development costs, but also reduces processing delays. In addition, unified data storage and access are achieved through the Paimon component, ensuring data consistency and providing data persistence capabilities.

[0047] By de-Hadoopizing, the resource monopoly of the Hadoop cluster is avoided, and computing and storage resources can be reused with other businesses, saving machine costs to a certain extent.

[0048] Combining the efficient data transmission capabilities of MQTT and Kafka components can improve the overall processing performance of the system.

[0049] The Superset component provides rich data visualization functions, making it easy for business personnel to monitor and analyze data in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 is a schematic diagram of a process flow of integrated flow and batch data in a vehicle according to some exemplary embodiments;

[0052] Figure 2 It is a flow chart of a method for processing integrated flow and batch data in a vehicle provided by an embodiment of the present application;

[0053] Figure 3 It is a flow chart of a method for processing integrated flow and batch data in a vehicle provided by an embodiment of the present application;

[0054] Figure 4 This is an architecture diagram of a system for processing integrated flow and batch data in a vehicle provided by an embodiment of the present application;

[0055] Figure 5 is a schematic diagram of the format of real-time data provided by an embodiment of the present application;

[0056] Figure 6 It is a schematic diagram of a process flow of integrated flow and batch data in a vehicle provided by an embodiment of the present application;

[0057] Figure 7 It is a structural block diagram of a system for processing integrated batch and flow data in a vehicle provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0059] The following is an explanation of the terms used in this application:

[0060] (1) Kafka: Kafka is a distributed messaging system suitable for high-throughput messaging. It splits, isolates, and backs up data through mechanisms such as topics, partitions, and replicas, ensuring consistency and efficiency in the data transmission process.

[0061] (2) Flink: Flink is a stream processing framework used for processing and analyzing integrated stream and batch data. It supports multiple programming modes such as Application Programming Interface (API) / Structured Query Language (SQL), and takes into account flexibility and ease of use.

[0062] (3) Paimon: Paimon is a data storage technology that supports the writing and reading of stream data and batch data, provides unified storage and efficient data access, and relies on storage such as S3 to provide data persistence capabilities.

[0063] (4) Superset: Superset is an open source data visualization tool used for data display and analysis.

[0064] like Figure 2 As shown, it shows a method flow chart of a method for processing integrated batch data in a vehicle provided by an embodiment of the present application. The method for processing integrated batch data in a vehicle may include:

[0065] Step 201: The Kafka component obtains real-time data reported by the vehicle.

[0066] The vehicle will obtain real-time vehicle data and surrounding environment data during driving, and send these data to the Kafka component in the form of data streams. The vehicle data includes at least the vehicle's position, speed, acceleration, heading angle, etc., and the surrounding environment data includes at least lane data, obstacles, traffic light data, etc.

[0067] Step 202: The Flink engine reads real-time data from the Kafka component, and writes the real-time data into the data service layer of the Paimon component in a streaming manner according to the first path.

[0068] The data service layer of the Paimon component can be a DWS layer or an Application Data Service (ADS) layer. In addition, the Paimon component can also include an ODS layer, a DWD layer, a DataWareHouse Middle (DWM) layer, etc., and multiple data layers can be selected according to actual needs to form the first path.

[0069] For example, the ODS layer, DWD layer, and DWS layer can be selected to form the first path, or the ODS layer, DWD layer, DWM layer, and DWS layer can be selected to form the first path, or the ODS layer, DWD layer, DWS layer, and ADS layer can be selected to form the first path, which is not limited in this embodiment.

[0070] The Flink engine can write real-time data in the form of data streams to each data layer node in the first path in sequence. Taking the first path including the ODS layer, DWD layer, and DWS layer as an example, the Flink engine writes the real-time data in the form of a data stream to the ODS layer, then reads the real-time data from the ODS layer, performs a first process on the real-time data, and writes the processed real-time data in the form of a data stream to the DWD layer; then reads the real-time data from the DWD layer, performs a second process on the real-time data, and writes the processed real-time data in the form of a data stream to the DWS layer.

[0071] Step 203: The data synchronization tool uses the Flink engine to obtain offline data provided by the external data source.

[0072] The data synchronization tool may call the Flink engine and obtain offline data from an external data source through the Flink engine. The data synchronization tool may be Seatunnel or other tools, which are not limited in this embodiment.

[0073] Step 204: The Flink engine writes the offline data in batches into the data service layer of the Paimon component according to the second path.

[0074] The second path is composed of various data layers in the Paimon component, and may be the same as the first path or different from the first path.

[0075] Step 205: The Flink engine writes the real-time data and offline data in the data service layer into the indicator library.

[0076] The indicator library can be a database, such as Doris database, MySQL database, etc.

[0077] After writing the real-time data and offline data into the indicator library, users can use analysis tools to query and analyze the real-time data and offline data in the indicator library in real time.

[0078] To sum up, the method for processing integrated stream and batch data in a vehicle provided by the embodiment of the present application is that the Flink engine writes the real-time data into the data service layer of the Paimon component in a streaming manner according to the first path; the Flink engine can write the offline data into the data service layer of the Paimon component in batches according to the second path; then, the Flink engine writes the real-time data and offline data in the data service layer into the indicator library. In this way, the Flink engine can be used to process integrated stream and batch data, which not only reduces the operation and maintenance cost, learning cost and development cost, but also reduces the processing delay, and the unified storage and access of data is realized through the Paimon component, which ensures the consistency of data and provides persistence capability for data.

[0079] By de-Hadoopizing, the resource monopoly of the Hadoop cluster is avoided, and computing and storage resources can be reused with other businesses, saving machine costs to a certain extent.

[0080] like Figure 3 As shown, it shows a flow chart of a method for processing integrated batch data in a vehicle provided by an embodiment of the present application. The method for processing integrated batch data in a vehicle may include:

[0081] Step 301: deploy the Flink engine on the Kubernetes cluster.

[0082] Kubernetes, abbreviated as K8S, is an open source containerized application for managing multiple hosts in a cloud platform. Kubernetes provides a mechanism for application deployment, planning, updating, and maintenance.

[0083] Step 302: Install the Paimon component and add the Jar package of the Paimon component to the directory of the Flink engine so that the Flink engine can access the Paimon component.

[0084] like Figure 4 As shown, the Flink engine is deployed on K8S, the Paimon component is deployed on S3 storage, and the Flink component can access the Paimon component to write data to the Paimon component or read data from the Paimon component.

[0085] Step 303: The Kafka component obtains the real-time data reported by the vehicle.

[0086] The vehicle will obtain real-time vehicle data and surrounding environment data during driving, and send these data to the Kafka component in the form of data streams. The vehicle data includes at least the vehicle's position, speed, acceleration, heading angle, etc., and the surrounding environment data includes at least lane data, obstacles, traffic light data, etc.

[0087] Specifically, the Kafka component obtains real-time data reported by vehicles, which may include:

[0088] (1) The Kafka component receives real-time data reported by vehicles based on the Message Queuing Telemetry Transport (MQTT) protocol.

[0089] Since MQTT's advantage lies in IoT data communication and its ability to persist messages is somewhat limited, we choose to use Kafka components as middleware to forward the real-time data reported by MQTT to the Kafka components for caching, making it easier for downstream businesses to consume data, thereby improving the overall throughput of the server and increasing the traceability period of data.

[0090] When reporting real-time data, you can use the Protobuf protocol to compress the real-time data. Protobuf is a data serialization protocol that is efficient and compact, and can effectively reduce the size of data packets during data transmission. Figure 5 A snippet of real-time data compressed using the Protobuf protocol is shown.

[0091] (2) The producer in the Kafka component sends real-time data to the broker according to the topic.

[0092] (3) The broker in the Kafka component stores real-time data according to topics.

[0093] Step 304: The Flink engine reads real-time data from the Kafka component, and writes the real-time data into the data service layer of the Paimon component in a streaming manner according to the first path.

[0094] The Flink engine reads real-time data from the Kafka component, including: the Flink engine as a consumer obtains real-time data of subscribed topics from the Kafka component.

[0095] After obtaining the real-time data, the Flink engine can write the real-time data into the data service layer of the Paimon component. Among them, the data service layer of the Paimon component can be a DWS layer or an ADS layer. In addition, the Paimon component can also include an ODS layer, a DWD layer, a DWM layer, etc., and can select multiple data layers required to form the first path according to actual needs. For example, the ODS layer, the DWD layer, and the DWS layer can be selected to form the first path, or the ODS layer, the DWD layer, the DWM layer, and the DWS layer can be selected to form the first path, or the ODS layer, the DWD layer, the DWS layer, and the ADS layer can be selected to form the first path, which is not limited in this embodiment.

[0096] The Flink engine can write real-time data in the form of data streams to each data layer node in the first path in sequence. Taking the first path including the ODS layer, DWD layer, and DWS layer as an example, the Flink engine writes the real-time data in the form of a data stream to the ODS layer, then reads the real-time data from the ODS layer, performs a first process on the real-time data, and writes the processed real-time data in the form of a data stream to the DWD layer; then reads the real-time data from the DWD layer, performs a second process on the real-time data, and writes the processed real-time data in the form of a data stream to the DWS layer.

[0097] Specifically, writing the real-time data into the data service layer of the Paimon component in a streaming manner according to the first path may include: for the first data layer node in the first path, the Flink engine writes the real-time data into the first data layer node; the Paimon component generates incremental data and data snapshots based on the real-time data; for the i-th data layer node in the first path, i≥2, the Flink engine reads the incremental data and data snapshots of the real-time data from the i-1-th data layer node, processes the real-time data accordingly based on the incremental data and data snapshots, and writes the processed real-time data into the i-th data layer node; the Paimon component generates incremental data and data snapshots based on the processed real-time data. The data layer node may be an ODS layer, a DWD layer, a DWM layer, a DWS layer, or an ADS layer. Different data layer nodes have different data processing methods.

[0098] The Flink engine uses the same process to write real-time data to each data layer node. The following uses writing real-time data to the first data layer node as an example to illustrate the data writing process.

[0099] Specifically, the Flink engine writes the real-time data to the first data layer node, which may include: if the real-time data is Protobuf format data stored in a key-value pair manner, the Flink engine converts the Protobuf format Value into a Json format Value; the Flink engine extracts information from the key and adds it to the format-converted Value; the Flink engine writes the added Value as real-time data to the first data layer node.

[0100] After converting the Protobuf format data into Json format data that is easier to query with SQL, the data is stored in the Catalog table of the Paimon component. Subsequent data processing links can use SQL to process real-time data.

[0101] Each data layer node includes a database and at least one data table, and writing real-time data to the data layer node means writing real-time data into the data table of the database. Specifically, the Flink engine writes real-time data into the first data layer node, which may include: the Flink engine reads the database information and data table information corresponding to the real-time data in the Piamon component; the Flink engine detects whether there is a database corresponding to the database information in the first data layer node; if there is a database corresponding to the database information, the Flink engine detects whether there is a data table corresponding to the data table information in the database; if there is a data table corresponding to the data table information, the Flink engine writes the real-time data into the data table; if there is no data table corresponding to the data table information, the Flink engine creates a data table in the database and writes the real-time data into the data table. If there is no database corresponding to the database information, the Flink engine creates a database in the first data layer node; the Flink engine detects whether there is a data table corresponding to the data table information in the database; if there is no data table corresponding to the data table information, the Flink engine creates a data table in the database and writes the real-time data into the data table.

[0102] Step 305: The data synchronization tool uses the Flink engine to obtain offline data provided by the external data source.

[0103] The data synchronization tool may call the Flink engine and obtain offline data from an external data source through the Flink engine. The data synchronization tool may be Seatunnel or other tools, which are not limited in this embodiment.

[0104] Step 306: The Flink engine writes the offline data in batches into the data service layer of the Paimon component according to the second path.

[0105] The second path is composed of various data layers in the Paimon component, and may be the same as the first path or different from the first path.

[0106] Specifically, the Flink engine writes offline data in batches to the data service layer of the Paimon component according to the second path, which may include: for the first data layer node in the second path, the Flink engine writes the offline data to the first data layer node; the Paimon component generates a data snapshot based on the offline data; for the i-th data layer node in the second path, i≥2, the Flink engine reads the data snapshot of the offline data from the i-1-th data layer node, performs corresponding processing on the offline data according to the data snapshot, and writes the processed offline data to the i-th data layer node; the Paimon component generates a data snapshot based on the processed offline data. Among them, the data layer node can be the ODS layer, DWD layer, DWM layer, DWS layer or ADS layer. Different data layer nodes have different data processing methods.

[0107] The Flink engine uses the same process to write offline data to each data layer node. The following uses writing offline data to the first data layer node as an example to illustrate the data writing process.

[0108] Specifically, when initializing the first data layer node, the Flink engine creates a database and data table corresponding to the offline data in the Paimon component; the Flink engine writes the offline data into the data table of the database.

[0109] Before writing offline data, you need to create a database and data table in the data layer node in advance. Then, the Flink engine can directly write the offline data into the data table without having to determine whether the database and data table exist during the writing process.

[0110] Step 307: The Flink engine writes the real-time data and offline data in the data service layer into the indicator library.

[0111] The indicator library can be a database, such as Doris database, MySQL database, etc.

[0112] Step 308: The Superset component reads real-time data and offline data from the indicator library, and generates a query report or a data analysis report for the real-time data and offline data.

[0113] After writing the real-time data and offline data into the indicator library, users can use analysis tools to query and analyze the real-time data and offline data in the indicator library in real time.

[0114] The Superset component is a data visualization tool that can provide real-time dashboard query and data analysis functions.

[0115] like Figure 6As shown, both the first path and the second path include the ODS layer, the DWD layer and the DWS layer. For real-time data, the operating vehicle sends the real-time data to the Flink engine through the Kafka component, and the Flink engine writes the real-time data to the DWS layer through the first path; for offline data, the external data source sends the offline data to the Flink engine through the data synchronization tool, and the Flink engine writes the offline data to the DWS layer through the second path.

[0116] To sum up, the method for processing integrated stream and batch data in a vehicle provided by the embodiment of the present application is that the Flink engine writes the real-time data into the data service layer of the Paimon component in a streaming manner according to the first path; the Flink engine can write the offline data into the data service layer of the Paimon component in batches according to the second path; then, the Flink engine writes the real-time data and offline data in the data service layer into the indicator library. In this way, the Flink engine can be used to process integrated stream and batch data, which not only reduces the operation and maintenance cost, learning cost and development cost, but also reduces the processing delay, and the unified storage and access of data is realized through the Paimon component, which ensures the consistency of data and provides persistence capability for data.

[0117] By de-Hadoopizing, the resource monopoly of the Hadoop cluster is avoided, and computing and storage resources can be reused with other businesses, saving machine costs to a certain extent.

[0118] Combining the efficient data transmission capabilities of MQTT and Kafka components can improve the overall processing performance of the system.

[0119] The Superset component provides rich data visualization functions, making it easy for business personnel to monitor and analyze data in real time.

[0120] like Figure 7 As shown, it shows a structural block diagram of a processing system for integrated stream and batch data in a vehicle provided by an embodiment of the present application. The processing system for integrated stream and batch data includes a Kafka component 710, a Flink engine 720, a Paimon component 730, a data synchronization tool 740, and a Superset component 750;

[0121] Kafka component 710, used to obtain real-time data reported by vehicles;

[0122] The Flink engine 720 is used to read real-time data from the Kafka component 710 and write the real-time data into the data service layer of the Paimon component 730 in a streaming manner according to the first path;

[0123] A data synchronization tool 740 is used to use the Flink engine 720 to obtain offline data provided by an external data source;

[0124] The Flink engine 720 is also used to write the offline data in batches into the data service layer of the Paimon component 730 according to the second path;

[0125] The Flink engine 720 is also used to write the real-time data and offline data in the data service layer into the database.

[0126] In an alternative embodiment,

[0127] For the first data layer node in the first path, the Flink engine 720 is further used to write the real-time data into the first data layer node; the Paimon component 730 is further used to generate incremental data and data snapshots according to the real-time data;

[0128] For the i-th data layer node in the first path, i≥2, the Flink engine 720 is also used to read the incremental data and data snapshots of the real-time data from the i-1-th data layer node, perform corresponding processing on the real-time data according to the incremental data and the data snapshot, and write the processed real-time data to the i-th data layer node; the Paimon component 73 is also used to generate incremental data and data snapshots according to the processed real-time data.

[0129] In an optional embodiment, the Flink engine 720 is further configured to:

[0130] If the real-time data is stored in Protobuf format using key-value pairs, convert the Protobuf format Value into a Json format Value;

[0131] Extract information from the key and add it to the converted value;

[0132] The supplemented Value is written into the first data layer node as real-time data.

[0133] In an optional embodiment, the Flink engine 720 is further configured to:

[0134] Read the database information and data table information corresponding to the real-time data in the Paimon component 730;

[0135] Check whether there is a database corresponding to the database information in the first data layer node;

[0136] If there is a database corresponding to the database information, then check whether there is a data table corresponding to the data table information in the database;

[0137] If there is a data table corresponding to the data table information, the real-time data is written into the data table.

[0138] In an alternative embodiment,

[0139] For the first data layer node in the second path, the Flink engine 720 is further used to write the offline data into the first data layer node; the Paimon component 730 is further used to generate a data snapshot according to the offline data;

[0140] For the i-th data layer node in the second path, i≥2, the Flink engine 720 is also used to read the data snapshot of the offline data from the i-1-th data layer node, perform corresponding processing on the offline data according to the data snapshot, and write the processed offline data to the i-th data layer node; the Paimon component 730 is also used to generate a data snapshot based on the processed offline data.

[0141] In an optional embodiment, the Flink engine 720 is further used to create a database and a data table corresponding to the offline data in the Paimon component 730 when initializing the first data layer node;

[0142] The Flink engine 720 is also used to write offline data into a data table of the database.

[0143] In an optional embodiment, the Kafka component 710 is further used to receive real-time data reported by the vehicle based on the MQTT protocol; the producer in the Kafka component 710 is used to send the real-time data to the broker according to the topic, and the broker in the Kafka component 710 is used to store the real-time data according to the topic;

[0144] The Flink engine 720 is also used as a consumer to obtain real-time data of subscribed topics from the Kafka component 710.

[0145] In an optional embodiment, the Superset component 750 is used to read real-time data and offline data from the indicator library, and generate a query report or a data analysis report for the real-time data and offline data.

[0146] In an optional embodiment, the processing system for integrated stream and batch data further includes a deployment module, which is used to:

[0147] Deploy the Flink engine 720 on the Kubernetes cluster;

[0148] Install Paimon component 730;

[0149] Add the Jar package of the Paimon component to the directory of the Flink engine 720 so that the Flink engine 720 can access the Paimon component 730 .

[0150] To sum up, in the system for processing integrated stream and batch data in a vehicle provided by the embodiment of the present application, the Flink engine writes the real-time data into the data service layer of the Paimon component in a streaming manner according to the first path; the Flink engine can write the offline data into the data service layer of the Paimon component in batches according to the second path; then, the Flink engine writes the real-time data and offline data in the data service layer into the indicator library, so that the Flink engine can be used to realize the processing of integrated stream and batch data, which not only reduces the operation and maintenance cost, learning cost and development cost, but also reduces the processing delay, and realizes unified storage and access of data through the Paimon component, ensures the consistency of data, and provides persistence capability for data.

[0151] By de-Hadoopizing, the resource monopoly of the Hadoop cluster is avoided, and computing and storage resources can be reused with other businesses, saving machine costs to a certain extent.

[0152] Combining the efficient data transmission capabilities of MQTT and Kafka components can improve the overall processing performance of the system.

[0153] The Superset component provides rich data visualization functions, making it easy for business personnel to monitor and analyze data in real time.

[0154] An embodiment of the present application provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the method for processing integrated flow and batch data in a vehicle as described above.

[0155] An embodiment of the present application provides a computer device, which includes a processing system for integrated flow and batch data in any of the above-mentioned vehicles.

[0156] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0157] The above description is not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. A method for processing integrated flow and batch data in a vehicle, characterized in that: The method comprises: The Kafka component obtains real-time data reported by vehicles; The Flink engine reads the real-time data from the Kafka component, and writes the real-time data into the data service layer of the Paimon component in a streaming manner according to the first path; The data synchronization tool uses the Flink engine to obtain offline data provided by the external data source; The Flink engine writes the offline data into the data service layer of the Paimon component in batches according to the second path; The Flink engine writes the real-time data and offline data in the data service layer into the indicator library.

2. The method for processing integrated flow and batch data in a vehicle according to claim 1, characterized in that: The step of writing the real-time data into the data service layer of the Paimon component in a streaming manner according to the first path includes: For the first data layer node in the first path, the Flink engine writes the real-time data into the first data layer node; the Paimon component generates incremental data and data snapshots according to the real-time data; For the i-th data layer node in the first path, i≥2, the Flink engine reads the incremental data and data snapshot of the real-time data from the i-1-th data layer node, performs corresponding processing on the real-time data according to the incremental data and the data snapshot, and writes the processed real-time data to the i-th data layer node; the Paimon component generates incremental data and data snapshots based on the processed real-time data.

3. The method for processing integrated flow and batch data in a vehicle according to claim 2, characterized in that: The Flink engine writes the real-time data into the first data layer node, including: If the real-time data is data in Protobuf format stored in a key-value pair manner, the Flink engine converts the Value in Protobuf format into a Value in Json format; The Flink engine extracts information from the key and adds it to the value after format conversion; The Flink engine writes the supplemented Value as real-time data into the first data layer node.

4. The method for processing integrated flow and batch data in a vehicle according to claim 2, characterized in that: The Flink engine writes the real-time data into the first data layer node, including: The Flink engine reads the database information and data table information corresponding to the real-time data in the Paimon component; The Flink engine detects whether there is a database corresponding to the database information in the first data layer node; If there is a database corresponding to the database information, the Flink engine detects whether there is a data table corresponding to the data table information in the database; If a data table corresponding to the data table information exists, the Flink engine writes the real-time data into the data table.

5. The method for processing integrated flow and batch data in a vehicle according to claim 1, characterized in that: The Flink engine writes the offline data in batches into the data service layer of the Paimon component according to the second path, including: For the first data layer node in the second path, the Flink engine writes the offline data into the first data layer node; the Paimon component generates a data snapshot according to the offline data; For the i-th data layer node in the second path, i≥2, the Flink engine reads the data snapshot of the offline data from the i-1-th data layer node, performs corresponding processing on the offline data according to the data snapshot, and writes the processed offline data to the i-th data layer node; the Paimon component generates a data snapshot based on the processed offline data.

6. The method for processing integrated flow and batch data in a vehicle according to claim 5, characterized in that: The method further comprises: when initializing the first data layer node, the Flink engine creates a database and a data table corresponding to the offline data in the Paimon component; The Flink engine writes the offline data into the first data layer node, including: the Flink engine writes the offline data into a data table of the database.

7. The method for processing integrated flow and batch data in a vehicle according to claim 1, characterized in that: The Kafka component obtains the real-time data reported by the vehicle, including: the Kafka component receives the real-time data reported by the vehicle based on the message queue telemetry transmission MQTT protocol; the producer in the Kafka component sends the real-time data to the agent according to the topic, and the agent in the Kafka component stores the real-time data according to the topic; The Flink engine reads the real-time data from the Kafka component, including: the Flink engine, as a consumer, obtains the real-time data of the subscribed topic from the Kafka component.

8. The method for processing integrated flow and batch data in a vehicle according to claim 1, characterized in that: The method further comprises: The Superset component reads the real-time data and the offline data from the indicator library, and generates a query report or a data analysis report for the real-time data and the offline data.

9. The method for processing integrated flow and batch data in a vehicle according to any one of claims 1 to 8, characterized in that: The method further comprises: Deploy the Flink engine on the Kubernetes cluster; Install the Paimon component; Add the Jar package of the Paimon component to the directory of the Flink engine so that the Flink engine can access the Paimon component.

10. A system for processing integrated data of flow and batch in vehicles, characterized in that: The stream-batch integrated data processing system includes Kafka components, Flink engine and data synchronization tools; The Kafka component is used to obtain real-time data reported by vehicles; The Flink engine is used to read the real-time data from the Kafka component and write the real-time data into the data service layer of the Paimon component in a streaming manner according to the first path; A data synchronization tool, used to use the Flink engine to obtain offline data provided by an external data source; The Flink engine is further used to write the offline data into the data service layer of the Paimon component in batches according to the second path; The Flink engine is also used to write the real-time data and offline data in the data service layer into the database.

Images