Traffic intelligent engine system construction method and device based on flow and batch integration

Abstract

This application discloses a method and device for constructing a traffic intelligent engine system based on stream-batch integration. First, a data link is constructed for each traffic device, and basic traffic data is imported into the StarRocks database through the data link; various data models of StarRocks are used to establish various data tables for data analysis; then, the analyzed data is imported into the target data table through the RoutineLoad method, and the target data table is managed for data storage using the storage and calculation separation method; finally, a preset monitoring and alarm scheme is used to monitor and alarm the target data table after data storage management. The present invention uses data stream and batch integration technology to achieve data fusion, thereby improving the processing efficiency of traffic data.

Description

Technical Field

[0001] The present invention relates to the field of big data traffic management technology, and in particular to a method and device for constructing a traffic intelligent engine system based on stream-batch integration. Background Art

[0002] Currently, technologies such as big data, artificial intelligence, and cloud computing are maturing in the transportation industry, and transportation services are constantly evolving. Urban transportation management faces two major challenges: a lack of efficient data collection methods and technologies, hindering real-time, dynamic, and refined data collection; and a lack of a unified data analysis platform and technologies to support the development of various transportation business applications.

[0003] In response to the above problems, existing technologies such as publication number CN116186173A specifically implement stream processing, batch processing, and fusion processing of big data through a big data storage and processing framework, while also achieving dynamic expansion of processing data and processing algorithms, and optimizing the entire system design by combining system design concepts such as OLAP, OLTP, and microservices.

[0004] However, existing technologies have low efficiency in processing traffic data, and cannot use data stream and batch integration technology to achieve data fusion. The efficiency of data extraction is low, the value of the data achieved is not high, and it cannot meet the needs of personnel on different business sides. Summary of the Invention

[0005] Based on this, the embodiments of the present application provide a method and device for constructing a traffic intelligent engine system based on flow and batch integration, which can solve the problems existing in the existing technology.

[0006] In a first aspect, a method for constructing a traffic intelligence engine system based on flow and batch integration is provided, the method comprising:

[0007] Build data links for various traffic devices and import basic traffic data into the StarRocks database through the data links. This includes: importing real-time data and offline data;

[0008] Use StarRocks' multiple data models to build various data tables for data analysis; wherein the data models include at least a detail model, an aggregation model, an update model, and a primary key model;

[0009] Import the analyzed data into the target data table through RoutineLoad;

[0010] Use storage and computing separation to manage data storage for target data tables. This includes building a unified storage access layer for data storage, using disk to cache hotspot data, and managing data through a pre-set data distribution method.

[0011] Use the preset monitoring and alarm scheme to monitor and alarm the target data table after data storage management.

[0012] Optionally, importing traffic equipment data according to the data link specifically includes:

[0013] Import offline data into StarRocks regularly through Broker Load; connect to message queues of different vendors and collect real-time data uniformly into Kafka.

[0014] Optionally, build data links for each traffic device and import basic traffic data into the StarRocks database through the data links, including:

[0015] Traffic data is checkpoint vehicle passing data, which is cleaned and converted according to business needs. The original checkpoint vehicle passing data is structured JSON data, and corresponding table fields are created according to the key and value types of JSON.

[0016] Optionally, the imported traffic data is analyzed according to the data model, including:

[0017] Create a table using the detail model and define a sort key when creating the table. The sort key is the camera ID. Include the defined sort key in the query filter conditions to quickly filter data.

[0018] Optionally, import the analyzed data into the target data table using RoutineLoad, including:

[0019] First, submit a SQL statement to StarRocks to create a RoutineLoad import job; then StarRocks automatically parses the SQL statement and creates a resident import job, continuously generating a series of import tasks; finally, StarRocks consumes messages from all or part of the partitions in the Kafka cluster Topic and imports them into the target data table in StarRocks.

[0020] Optionally, the analyzed data is imported into the target data table through RoutineLoad, which also includes:

[0021] A corresponding label is set in each import job; the label is specified by the user or automatically generated by the system and is used to identify the import job.

[0022] Optionally, data storage and computing separation is used to manage the target data table, including:

[0023] Back-end storage is performed through an object storage system compatible with the AWS S3 protocol and HDFS deployed in traditional data centers. Table data is uniformly stored in the bucket or HDFS directory specified by the object storage, and stored in subdirectories such as data / and meta / depending on the type.

[0024] Optionally, data management is performed using a preset data distribution method, including:

[0025] Data management is performed using the Hash data distribution method and the Range+Hash data distribution method. In the Hash data distribution method, a table is a partition, and the partitions further divide data according to the bucket key and the number of buckets. In the Range+Hash data distribution method, a table is split into multiple partitions, and each partition further divides data according to the bucket key and the number of buckets.

[0026] Optionally, a preset monitoring and alarm scheme is used to monitor and alarm the target data table after data storage management, including:

[0027] Use the built-in StarRocksManager or Prometheus+Grafana solution for monitoring and alarming. Among them: using the built-in StarRocksManager for monitoring and alarming, specifically including: using the built-in Agent to collect monitoring information from each Host, reporting it to the Center Service, and then making a visual display; using StarRocksManager to provide alarm notifications through email and webhooks; using the Prometheus+Grafana solution for monitoring and alarming, specifically including: obtaining cluster monitoring information by directly connecting to the HTTP port of BE or FE, Prometheus accessing the Metric interface of FE or BE through Pull, and then storing the monitoring data in a time series database.

[0028] In a second aspect, a device for constructing a traffic intelligent engine system based on flow and batch integration is provided, the device comprising:

[0029] The data aggregation module is used to establish a data link with each traffic device and import traffic data into the StarRocks database through the data link; specifically, it includes: importing real-time data and offline data;

[0030] A data modeling module is used to construct multiple data models and establish various data tables for data analysis in the data models; wherein the data models include at least a detail model, an aggregation model, an update model, and a primary key model;

[0031] The data integration module is used to determine the resident target data table and import the analyzed data into the target data table through the RoutineLoad method;

[0032] The data storage management module is used to manage the data storage of the target data table by separating storage and computing. Specifically, it includes building a unified storage access layer for data storage, using disk to cache hot data, and managing data through a preset data distribution method.

[0033] The system monitoring and operation and maintenance module is used to monitor and alarm the target data table after data storage management using a preset monitoring and alarm solution.

[0034] The technical solution provided in the embodiment of this application first establishes a data link with each traffic device, and imports traffic data into the StarRocks database through the data link; constructs multiple data models, and establishes various data tables for data analysis in the data models; then determines the resident target data table, and imports the data into the target data table through the RoutineLoad protocol; uses the storage and calculation separation method to perform data storage management on the target data table; and finally uses a preset monitoring and alarm solution to monitor and alarm the target data table after data storage management. It can be seen that the beneficial effects of the present invention include at least:

[0035] (1) Simple distributed architecture. By adopting a simple distributed architecture, it can be horizontally expanded to handle large-scale data and high-concurrency queries. Data is sharded and stored on multiple nodes, separating storage and computing, enabling parallel processing and querying of data, improving system scalability, and reducing costs.

[0036] (2) High-performance query. StarRocks provides excellent query performance due to its columnar storage, distributed architecture, and vectorized engine. It supports a variety of query operations, including aggregation, sorting, joining, and filtering, making it suitable for complex analytical queries and supporting high QPS.

[0037] (3) Flexible data model. It supports multi-dimensional data models, which facilitates multi-dimensional analysis. It supports a variety of data types and flexible data model design, which is suitable for various data analysis needs. It also supports rapid update and deletion in massive data scenarios.

[0038] (4) Easy to use and manage. It provides an easy-to-use management interface and command line tools to facilitate user configuration, monitoring, and management of the cluster. It also supports resource isolation, making it easier to locate and resolve user issues. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can, without inventive effort, derive other implementation drawings based on the provided drawings.

[0040] Figure 1 An overall flow chart of a method for constructing a traffic intelligence engine system based on flow-batch integration provided in an embodiment of the present application;

[0041] Figure 2 A flowchart of the data integration part provided in the embodiment of this application;

[0042] Figure 3 A diagram of the monitoring architecture provided in an embodiment of the present application;

[0043] Figure 4 A block diagram of a traffic intelligent engine system construction device based on flow and batch integration provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of this application clearer, this application is further described in detail below with reference to the accompanying drawings and embodiments.

[0045] In the description of the present invention, the terms "comprise", "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may also include other steps or units that are not explicitly listed but are inherent to these processes, methods, products or apparatuses, or steps or units that are added based on further optimization solutions conceived by the present invention.

[0046] Stream-batch integration refers to a data processing method that combines stream processing and batch processing. Traditionally, data processing is divided into two modes: stream processing mode and batch processing mode.

[0047] In stream processing mode, data arrives in real time and is processed immediately without waiting for a certain time window.

[0048] In batch mode, data is collected over a period of time and processed all at once. A common example is performing statistical analysis on daily sales data.

[0049] Stream-batch integration resolves the conflict between real-time performance and accuracy by combining batch and stream processing. It can process real-time data streams and perform operations such as aggregation, filtering, and transformation on the data as needed. It can also process historical data for offline analysis. It can be applied in various scenarios, such as real-time monitoring systems, real-time risk control systems, and real-time recommendation systems. It provides more flexible and efficient data processing capabilities, meeting the needs of applications with high real-time requirements.

[0050] Existing technologies are inefficient in processing traffic data and cannot leverage data stream and batch integration for data fusion. This results in inefficient data extraction, low data value, and a failure to meet the needs of diverse business personnel. Furthermore, existing engines are unable to meet the requirements for high-QPS scenarios, complex queries, and joins, resulting in insufficient query performance. Furthermore, data from different vendors in the traffic system is fragmented and stored separately. Different vendors utilize different technologies, resulting in a multitude of technical components, making maintenance and development difficult.

[0051] Based on this, this application proposes a method for building a traffic intelligent engine system based on flow and batch integration to solve the above problems. Please refer to Figure 1 , the method may include the following steps:

[0052] Step 101: Build data links for each traffic device and import basic traffic data into the StarRocks database through the data links. This includes importing both real-time and offline data. This step implements the data aggregation function, which includes:

[0053] Connect the data links and upstream and downstream ecosystems of transportation equipment from different manufacturers, including automated real-time and offline data import; build a cloud-native management and control platform to provide efficient operation and maintenance management and business delivery capabilities.

[0054] Offline data is regularly imported into StarRocks through Broker Load, and real-time data is connected to message queues of different vendors and collected uniformly into Kafka.

[0055] Step 102: Use StarRocks' various data models to create various data tables for data analysis. This step implements the data modeling function, including:

[0056] This embodiment mainly supports four data models: the Duplicate Key Model, the Aggregate Key Model, the Unique Key Model, and the Primary Key Model. These four data models can support a variety of data analysis scenarios, such as log analysis, data summary analysis, and real-time analysis. The following describes the establishment process and usage of the four models in conjunction with data table design:

[0057] (1) Duplicate Key Model

[0058] Establishment process: Treat each piece of raw data as a record and retain all dimension and indicator information.

[0059] Application: Suitable for scenarios requiring detailed analysis of raw data, such as log analysis and anomaly detection. Through the detailed model, you can directly query and analyze raw data to obtain more detailed and accurate analysis results.

[0060] Table Design: Traffic system data scenario analysis data (traffic data) primarily consists of vehicle passing data at checkpoints. This data is immutable and is appended to existing data, with no modification of historical data. Therefore, a detailed model (Duplicate Key Model) is used to create the table. When creating the table, a sort key is defined, which is the checkpoint camera ID. By including the sort key in query filter conditions, data can be quickly filtered, improving query efficiency.

[0061] (2) Aggregate KeyModel

[0062] Establishment process: Aggregate the raw data to generate summary data. Select appropriate dimensions and indicators for aggregation based on business needs.

[0063] Application: Suitable for scenarios requiring data aggregation and analysis, such as daily road traffic statistics and trend analysis. Aggregation models can reduce data volume and improve query efficiency while still meeting common analytical needs.

[0064] Table Design: Traffic system data scenario analysis data includes traffic indicators, which are mostly queried using specific dimensions. Old data is infrequently updated, with only new data appended. For this reason, the Aggregate Key Model is used to create tables. When creating the table, a sort key is defined, which is the road segment ID. From data import to data query, data with the same sort key is aggregated multiple times within the Aggregate Key Model. This reduces the amount of data required for query processing, thereby improving query efficiency.

[0065] (3) Update Model (Unique Key Model)

[0066] Creation Process: When creating a table, you can define primary keys (such as checkpoint ID and time) and indicator columns. Querying a table returns the latest data from a set of data with the same primary key. Compared to the detailed model, the update model simplifies the data import process and better supports scenarios with real-time and frequent updates.

[0067] Application: Suitable for scenarios requiring real-time analysis and updates, such as real-time trend statistics of vehicle passing data at checkpoints. By updating the model, the latest data information can be obtained in a timely manner, and real-time analysis and decision-making can be carried out.

[0068] (4) Primary Key Model

[0069] Establishment process: Associate and organize related data based on primary keys (such as checkpoint ID, road section ID, etc.) to form a hierarchical data model.

[0070] Purpose: Connect transactional data to StarRocks in real time, such as road section data and checkpoint data.

[0071] Table Design: Traffic system data scenario analysis includes road network data. In addition to inserting data, it typically involves numerous update and delete operations. For this reason, a primary key model is used for table creation. When creating the table, a primary key is defined, which is the road segment ID. This eliminates the need for aggregation operations during queries, ensuring efficient query execution.

[0072] Step 103, import the analyzed data into the target data table through RoutineLoad. In this step, the data integration function is specifically implemented, such as Figure 2 , specifically including:

[0073] Data import cleans, converts, and loads the original checkpoint vehicle passing data into StarRocks according to business needs, allowing for extremely fast and unified data analysis in the StarRocks system. Data import can import multiple data sources, such as offline vehicle passing data, real-time message data, etc. Data import is achieved through import jobs. Each import job has a label (Label), which is specified by the user or automatically generated by the system to identify the import job. Each label is unique within a database and can only be used for a successful import job. After an import job is successful, its label can no longer be used to submit other import jobs. Only the labels of failed import jobs can be used to submit other import jobs. This mechanism ensures that the data corresponding to any label is imported at most once, that is, it implements the "at-most-once" semantics.

[0074] All import methods provide atomicity guarantees, meaning that all valid data within the same import job is either fully effective or completely ineffective, ensuring that only partial data is not imported. This does not include data that is filtered out due to data quality issues such as type conversion errors.

[0075] Among them, in the optional embodiments of this application, specifically including:

[0076] (1) Specify the target database and table

[0077] By creating a data table, specify the created table;

[0078] (2) Create a permanent import job SQL

[0079] To continuously import Kafka data streams into StarRocks using RoutineLoad, first submit a SQL statement to StarRocks to create a RoutineLoad import job. StarRocks then automatically parses the SQL statement and creates a resident import job, continuously generating a series of import tasks. Finally, StarRocks consumes messages from all or some partitions of a Kafka cluster topic and imports them into the target data table in StarRocks.

[0080] (3) SQL to execute the import job

[0081] Data from a Kafka message queue can be imported into the traffic_cars table in the traffic_db database. Specifically, a RoutineLoad task is created in the traffic_cars table in the traffic_db database. This task consumes data from the specified Kafka message queue and loads the data into the traffic_cars table using the specified columns and delimiter. Properties such as the number of concurrent loading tasks, the list of Kafka brokers, the Kafka topic, and the list of Kafka partitions are specified to control the behavior and configuration of the import process. During the import process, specific data columns are selectively imported, and a column named pay_dt is generated through expression calculation. Properties such as the number of concurrent loading tasks, the JSON data format, and the path to each field in the JSON are also specified. In summary, a RoutineLoad import job named traffic is submitted to StarRocks to continuously consume messages from the topic named traffic_data in the Kafka cluster and import them into the traffic_cars table in the traffic_db database. The import job will start consuming from the earliest offset of the partition specified by the topic.

[0082] Step 104: Manage the target data table's data storage using a storage-computation separation approach. This includes building a unified storage access layer for data storage, using disks to cache hotspot data, and managing data using a pre-set data distribution method. This step implements the data storage management function, specifically:

[0083] Data storage utilizes a storage-computing separation technology solution. The core of this solution is to decouple storage from computing, making compute nodes stateless and enabling rapid scaling. Furthermore, object storage generally offers better data reliability and lower costs. Data storage utilizes columnar storage and employs a partitioning and bucketing mechanism for data management.

[0084] (1) Storage

[0085] Build a unified storage access layer to shield the differences between different back-end storage systems. Based on this unified storage access layer, this system can easily support new storage systems while maintaining transparency to upper-layer applications. Support a variety of mainstream storage systems, including compatible object storage systems (such as OSS, MinIO) and HDFS. Currently, the storage and computing separation technology of this system supports two back-end storage methods: object storage systems compatible with the AWSS3 protocol (supported by mainstream object storage) and HDFS deployed in traditional data centers. This technology stores table data in a unified manner in the bucket or HDFS directory specified by the object storage, and stores it in sub-directories such as data / and meta / depending on the type. In the above step 2, the data modeling module can specify the storage system, or use the internal storage system.

[0086] (2) Cache

[0087] To mitigate performance impacts, hot data is cached on local disk. Users can choose to enable hot data caching and specify a hot data lifecycle when creating a table. From the time data is written to the end of its lifecycle, hot data will be cached on local disk to improve query efficiency.

[0088] (3) Data management

[0089] The following two data distribution modes are supported:

[0090] Hash data distribution: Each table is divided into partitions, which are further divided into bucket keys and bucket counts. This method splits a table into different management units based on the partition key, and selects a corresponding storage strategy for each management unit, such as the number of replicas, number of buckets, hot / cold strategies, and storage media. Multiple storage media can be used within a cluster. Partitions containing new data can be placed on SSDs, leveraging the SSD's excellent random read / write performance to improve query performance, while older data can be stored on SATA drives to reduce data storage costs.

[0091] Range+Hash data distribution: A table is split into multiple partitions, each of which further divides data based on bucket keys and bucket counts. Within a partition, data with the same bucket key hash value forms a tablet. Tablets are stored in redundant replicas and serve as the smallest unit for data balancing and recovery. Tablet replicas are managed by a separate local storage engine, and data import and query operations are ultimately performed on the relevant tablet replicas.

[0092] Step 105: Use the preset monitoring and alarm scheme to monitor and alarm the target data table after data storage management. In this step, the system monitoring and operation function is specifically implemented, and two monitoring and alarm schemes are specifically provided:

[0093] (1) Use the built-in StarRocksManager, such as Figure 3 As shown in the figure, its built-in agent collects monitoring information from each host, reports it to the Center Service, and then displays it visually. StarRocksManager provides alarm notifications via email and webhooks. StarRocksManager monitoring can be divided into two dimensions: cluster and node.

[0094] (2) Provide Prometheus+Grafana solution architecture diagram as follows Figure 3 As shown, StarRocks provides a Prometheus-compatible information collection interface. You can obtain cluster monitoring information by directly connecting to the HTTP port of the BE or FE. Prometheus accesses the FE or BE's Metric interface through a pull request and then stores the monitoring data in a time series database. Users can configure Prometheus as a data source in Grafana and draw custom dashboards.

[0095] like Figure 4 The embodiment of the present application also provides a device for constructing a traffic intelligent engine system based on flow and batch integration, the device comprising:

[0096] The data aggregation module is used to establish data links with various traffic devices and import traffic data into the StarRocks database through the data links; specifically, it includes:

[0097] Import of real-time data and offline data;

[0098] A data modeling module is used to construct multiple data models and establish various data tables for data analysis in the data models; wherein the data models include at least a detail model, an aggregation model, an update model, and a primary key model;

[0099] The data integration module is used to determine the resident target data table and import the analyzed data into the target data table through the RoutineLoad method;

[0100] The data storage management module is used to manage the data storage of the target data table by separating storage and computing. Specifically, it includes: building a unified storage access layer for data storage, using disk to cache hot data, and managing data through a preset data distribution method;

[0101] The system monitoring and operation and maintenance module is used to monitor and alarm the target data table after data storage management using a preset monitoring and alarm solution.

[0102] The traffic intelligent engine system construction device based on flow and batch integration provided in the embodiment of the present application is used to implement the above-mentioned traffic intelligent engine system construction method based on flow and batch integration. Regarding the specific limitations of the traffic intelligent engine system construction device based on flow and batch integration, please refer to the limitations of the traffic intelligent engine system construction method based on flow and batch integration in the above text, which will not be repeated here. The various parts of the above-mentioned traffic intelligent engine system construction device based on flow and batch integration can be fully or partially implemented by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the device in the form of hardware, or can be stored in the memory in the device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0103] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0104] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for constructing a traffic intelligent engine system based on flow and batch integration, characterized in that: Methods include: Build data links between various traffic devices and import basic traffic data into the StarRocks database through the data links, including the import of real-time data and offline data; Use StarRocks' multiple data models to build various data tables for data analysis; wherein the data models include at least a detail model, an aggregation model, an update model, and a primary key model; Import the analyzed data into the target data table through RoutineLoad; Use storage and computing separation to manage data storage for target data tables. This includes building a unified storage access layer for data storage, using disk to cache hotspot data, and managing data through a pre-set data distribution method. Use the preset monitoring and alarm scheme to monitor and alarm the target data table after data storage and management; Import the analyzed data into the target data table through RoutineLoad, including: First, submit a SQL statement to StarRocks to create a RoutineLoad import job. StarRocks then automatically parses the SQL statement and creates a resident import job, continuously generating a series of import tasks. Finally, StarRocks consumes messages from all or some partitions in the Kafka cluster topic and imports them into the target data table in StarRocks. The target data table is managed using a storage and computing separation approach, including: Back-end storage is performed using an object storage system compatible with the AWS S3 protocol and HDFS deployed in traditional data centers. Table data is uniformly stored in a bucket or HDFS directory specified by the object storage, and stored in the data / and meta / subdirectories depending on the data type. Data management is performed through pre-set data distribution methods, including: Data management is performed using the Hash data distribution method and the Range+Hash data distribution method. In the Hash data distribution method, a table is a partition, and the partitions further divide data according to the bucket key and the number of buckets. In the Range+Hash data distribution method, a table is split into multiple partitions, and each partition further divides data according to the bucket key and the number of buckets.

2. The method for constructing a traffic intelligence engine system according to claim 1, characterized in that: Importing traffic equipment data according to the data link specifically includes: Import offline data into StarRocks regularly through Broker Load; connect to message queues of different vendors and collect real-time data uniformly into Kafka.

3. The method for constructing a traffic intelligence engine system according to claim 1, characterized in that: Build data links for various traffic devices and import basic traffic data into the StarRocks database through the data links, including: Traffic data is checkpoint vehicle passing data, which is cleaned and converted according to business needs. The original checkpoint vehicle passing data is structured JSON data, and corresponding table fields are created according to the key and value types of JSON.

4. The method for constructing a traffic intelligence engine system according to claim 1, characterized in that: Analyze the imported traffic data according to the data model, including: Create a table using the detail model and define a sort key when creating the table. The sort key is the camera ID. Include the defined sort key in the query filter conditions to quickly filter data.

5. The method for constructing a traffic intelligence engine system according to claim 1, characterized in that: Import the analyzed data into the target data table through RoutineLoad, which also includes: A corresponding label is set in each import job; the label is specified by the user or automatically generated by the system and is used to identify the import job.

6. The method for constructing a traffic intelligence engine system according to claim 1, characterized in that: Use the preset monitoring and alarm scheme to monitor and alarm the target data table after data storage and management, including: Use the built-in StarRocksManager or Prometheus+Grafana solution for monitoring and alarming. Among them: using the built-in StarRocksManager for monitoring and alarming, specifically including: using the built-in Agent to collect monitoring information from each Host, reporting it to the Center Service, and then making a visual display; using StarRocksManager to provide alarm notifications through email and webhooks; using the Prometheus+Grafana solution for monitoring and alarming, specifically including: obtaining cluster monitoring information by directly connecting to the HTTP port of BE or FE, Prometheus accessing the Metric interface of FE or BE through Pull, and then storing the monitoring data in a time series database.

7. A traffic intelligent engine system construction device based on flow and batch integration, characterized in that: The device comprises: The data aggregation module is used to establish a data link with each traffic device and import traffic data into the StarRocks database through the data link; specifically, it includes: importing real-time data and offline data; A data modeling module is used to construct multiple data models and establish various data tables for data analysis in the data models; wherein the data models include at least a detail model, an aggregation model, an update model, and a primary key model; The data integration module is used to determine the resident target data table and import the analyzed data into the target data table through the RoutineLoad method; The data storage management module is used to manage the data storage of the target data table by separating storage and computing. Specifically, it includes: building a unified storage access layer for data storage, using disk to cache hot data, and managing data through a preset data distribution method; The system monitoring and operation module is used to monitor and alarm the target data table after data storage and management using the preset monitoring and alarm scheme; Import the analyzed data into the target data table through RoutineLoad, including: First, submit a SQL statement to StarRocks to create a RoutineLoad import job. StarRocks then automatically parses the SQL statement and creates a resident import job, continuously generating a series of import tasks. Finally, StarRocks consumes messages from all or some partitions in the Kafka cluster topic and imports them into the target data table in StarRocks. The target data table is managed using a storage and computing separation approach, including: Back-end storage is performed using an object storage system compatible with the AWS S3 protocol and HDFS deployed in traditional data centers. Table data is uniformly stored in a bucket or HDFS directory specified by the object storage, and stored in the data / and meta / subdirectories depending on the data type. Data management is performed through pre-set data distribution methods, including: Data management is performed using the Hash data distribution method and the Range+Hash data distribution method. In the Hash data distribution method, a table is a partition, and the partitions further divide data according to the bucket key and the number of buckets. In the Range+Hash data distribution method, a table is split into multiple partitions, and each partition further divides data according to the bucket key and the number of buckets.

Images