ClickHouse-based fused CDN (Content Delivery Network) service data processing method, device and equipment

Abstract

The invention discloses a ClickHouse-based fused CDN (Content Delivery Network) business data processing method, device and equipment, and the method comprises the steps: obtaining business data of a corresponding fused CDN manufacturer through calling a data interface of each fused CDN manufacturer in real time; performing preprocessing including data format standardization, field expansion and invalid data filtering on the service data by using Spark, packaging the preprocessed service data and sending the packaged service data to a Kafka cluster; the method comprises the following steps: creating a distributed data table based on a ReplacingMergeTree engine in a ClickHouse database, consuming service data in a Kafka cluster in real time by utilizing Spark, and writing the service data into the distributed data table of the ClickHouse database in batches; and retrieving target business data from the ClickHouse database according to a data query request of a user, and converting the target business data into a JSON format to serve as a query result to be returned. The query performance, the storage efficiency, the expandability and the write-in performance can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device and equipment for processing fusion CDN business data based on ClickHouse. Background Art

[0002] With the rapid development of Internet technology, especially the continuous expansion of content distribution network (CDN) services, higher requirements are placed on the performance, storage efficiency, scalability and write performance of data processing systems. However, the performance of the currently widely used traditional relational databases (such as MySQL, PostgreSQL, etc.) will significantly decrease when processing large-scale integrated CDN business data, especially when performing complex multi-dimensional queries or aggregate analysis. Since these databases are mainly based on row storage models, when certain fields need to be extracted from large data sets, the entire row of data needs to be read, resulting in increased I / O operations and long query response time. However, the traditional row storage method bundles all field data together and lacks optimized storage strategies for different types of data, resulting in low data compression ratios and serious waste of storage space. In CDN services, the data scale is often measured in PBs. Inefficient storage methods not only increase hardware costs, but also limit the flexibility and efficiency of data processing.

[0003] Traditional databases have poor scalability and are prone to performance bottlenecks when processing large-scale data in a single-machine deployment. Even if they are expanded through vertical expansion (increasing hardware resources) or horizontal sharding (sharding databases and tables), the operation is complex and costly, and it is difficult to guarantee the performance and stability of the system after expansion.

[0004] Low write performance is a significant disadvantage of traditional databases when handling high-throughput write operations. Frequent data updates and insertions in CDN services require database systems to have efficient write capabilities. However, traditional databases often increase data latency when handling such operations, affecting the real-time and accuracy of the business. Summary of the invention

[0005] In view of this, the purpose of the present invention is to propose a method, device and equipment for processing integrated CDN business data based on ClickHouse, aiming to solve the problems of low query performance, low storage efficiency, poor scalability and low write performance when traditional databases process large-scale integrated CDN business data.

[0006] To achieve the above object, the present invention provides a method for processing fusion CDN business data based on ClickHouse, the method comprising:

[0007] By calling the data interfaces of each integrated CDN vendor in real time, the business data of the corresponding integrated CDN vendor can be obtained;

[0008] Using Spark to preprocess the business data, including data format standardization, field expansion, and filtering of invalid data, and packaging the preprocessed business data and sending it to the Kafka cluster;

[0009] By creating a distributed data table based on the ReplacingMergeTree engine in the ClickHouse database, using Spark to consume the business data in the Kafka cluster in real time, and writing the business data in batches into the distributed data table of the ClickHouse database;

[0010] According to the user's query data request, the target business data is retrieved from the ClickHouse database, and the target business data is converted into JSON format and returned as the query result.

[0011] Preferably, the data types of the business data include bandwidth request data, back-to-source bandwidth request data, status code data, regional bandwidth request data, and TopURL data; and the business data of the corresponding fusion CDN vendors is acquired by calling the data interfaces of each fusion CDN vendor in real time, including:

[0012] Initialize the system by loading the system configuration file, initialize the acquisition task of each data type, and establish a communication connection with the Kafka cluster;

[0013] Traverse the fusion CDN vendor list according to the configuration file, traverse the domain name list of each fusion CDN vendor based on the fusion CDN vendor list, and obtain the domain name data of the target domain name;

[0014] Retrieve the collection time point information of each target domain name from the MySQL database, obtain the business data of the specified collection time point information by calling the data interface of the corresponding integrated CDN vendor, convert it into JSON format and cache it in the system memory;

[0015] The business data in JSON format cached in the system memory is transferred in batches to the Kafka cluster, and the next collection time point information of the target domain name in the MySQL database is updated to obtain the next round of business data.

[0016] Preferably, the using of Spark to perform preprocessing on the business data including data format standardization, field expansion, and filtering of invalid data, and packaging the preprocessed business data and sending it to the Kafka cluster includes:

[0017] Use Spark to set the Kafka cluster address, port, and Kafka Topic information, and deploy Spark tasks in a distributed environment. After the Spark tasks are initialized, consume the business data in the Kafka cluster in real time.

[0018] By traversing the received business data and parsing the business data in JSON format one by one, the business data that fails to be parsed or does not meet the requirements in format are filtered as invalid data;

[0019] The filtered business data is cached in the local memory queue. When the amount of data in the local memory queue reaches the data volume threshold, the current batch of business data is batch-transferred to the Kafka cluster by triggering the batch sending mechanism.

[0020] Preferably, the method creates a distributed data table based on the ReplacingMergeTree engine in the ClickHouse database, consumes the business data in the Kafka cluster in real time using Spark, and writes the business data in batches into the distributed data table of the ClickHouse database, including:

[0021] By creating a distributed data table based on the ReplacingMergeTree engine in the ClickHouse database, Spark is used to consume the business data in the Kafka cluster in real time and cache it in the local memory queue;

[0022] When the amount of business data in the local memory queue reaches the data amount threshold, the business data is written in batches to the distributed data table of the ClickHouse database.

[0023] Preferably, after writing the business data in batches into a distributed data table of the ClickHouse database, further comprising:

[0024] The Kafka Offset update mechanism is used to automatically update the consumption offset of the Kafka cluster to track the consumption progress of the business data.

[0025] Preferably, before retrieving the target business data from the ClickHouse database according to the user's query data request and converting the target business data into JSON format and returning it as a query result, it includes:

[0026] The metadata information of the distributed data table in the ClickHouse database is set through the configuration file, and the metadata information includes address information and table information.

[0027] Preferably, the step of retrieving target business data from the ClickHouse database according to the user's query data request and converting the target business data into JSON format to return as a query result includes:

[0028] Initialize the ClickHouse database, and according to the data type queried in the query data request, call the corresponding query logic to retrieve data from the distributed data tables of different ClickHouse databases in turn to obtain the target business data;

[0029] The target business data is formatted in JSON using a data rendering mechanism, and the processed query results are returned to the front end.

[0030] To achieve the above object, the present invention also provides a fusion CDN business data processing device based on ClickHouse, the device comprising:

[0031] A data acquisition unit, used to acquire the business data of the corresponding fusion CDN vendor by calling the data interface of each fusion CDN vendor in real time;

[0032] A data preprocessing unit, used to use Spark to preprocess the business data, including data format standardization, field expansion, and filtering of invalid data, and to package the preprocessed business data and send it to the Kafka cluster;

[0033] A data storage unit is used to create a distributed data table based on the ReplacingMergeTree engine in the ClickHouse database, consume the business data in the Kafka cluster in real time using Spark, and write the business data in batches into the distributed data table of the ClickHouse database;

[0034] The data query unit is used to retrieve target business data from the ClickHouse database according to the user's query data request, and convert the target business data into JSON format and return it as a query result.

[0035] In order to achieve the above-mentioned objectives, the present invention also proposes a fusion CDN business data processing device based on ClickHouse, comprising a processor, a memory and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of a fusion CDN business data processing method based on ClickHouse as described in the above-mentioned embodiment.

[0036] In order to achieve the above-mentioned purpose, the present invention also proposes a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the steps of a ClickHouse-based fusion CDN business data processing method as described in the above embodiment.

[0037] In order to achieve the above-mentioned objectives, the present invention also proposes a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of a ClickHouse-based fusion CDN business data processing method as described in the above-mentioned embodiment.

[0038] Beneficial effects:

[0039] The above solution can ensure the timeliness and accuracy of data by calling the data interface of CDN manufacturers in real time, meeting the business's demand for real-time data. Using Spark for data preprocessing effectively improves data quality, reduces errors and exceptions in subsequent processing, and improves the overall efficiency and accuracy of data processing; and realizes reliable data transmission through Kafka cluster, laying the foundation for subsequent processing. Using ClickHouse's ReplacingMergeTree engine to create distributed data tables not only optimizes the data storage structure and query performance, but also improves the overall efficiency of data processing, and significantly improves data writing performance through distributed deployment and batch writing mechanisms. At the same time, ClickHouse can use more efficient compression algorithms for data compression, which can greatly reduce the amount of data storage and data transmission, and the column storage characteristics of ClickHouse greatly improve query performance, especially for multi-dimensional queries and aggregate analysis, which can greatly improve data query performance and return results.

[0040] The above solution can support data acquisition from multiple CDN vendors and domain names through the initialization of the configuration file, enhancing the diversity and flexibility of the data source; managing the collection tasks and data points through the MySQL database realizes the automation and systematization of data collection and reduces manual intervention; using the system memory to cache data in JSON format reduces disk I / O and improves the efficiency of data transmission.

[0041] The above solution uses Spark's powerful data processing capabilities to achieve real-time consumption and preprocessing of data in the Kafka cluster. It improves data quality by parsing and filtering invalid data, avoiding errors and exceptions in subsequent processing. It adopts a batch sending mechanism to reduce the number of network transmissions and improve data transmission efficiency.

[0042] The above solution improves the efficiency and scalability of data storage by creating a distributed data table based on the ReplacingMergeTree engine. It also uses the Kafka Offset update mechanism to automatically track the progress of business data consumption, ensuring the continuity and integrity of data processing.

[0043] The above solution sets the metadata information of the ClickHouse database table through the configuration file, which improves the flexibility and configurability of the query and facilitates rapid adjustment according to different business needs; according to the data type in the query data request, the corresponding query logic is called for data retrieval, which improves the accuracy and efficiency of the query, and uses the data rendering mechanism to format the query results in JSON and then return them, which is convenient for front-end display and subsequent processing, and improves the user experience and the convenience of data application.

[0044] The above solution adds more nodes based on ClickHouse for horizontal expansion. Each node can process a part of the data and query requests. This method can be expanded as demand grows. The entire data processing process is based on a distributed architecture design and has good scalability and stability. Whether it is data collection, processing or storage query, business growth and changes can be coped with by adding nodes or optimizing resource allocation. The Spark Streaming open source big data framework is used to consume the integrated CDN business data of the Kafka queue in real time, and batch data is written to the ClickHouse data warehouse. The data is written after being processed in batches, which can reduce write overhead and improve database write performance, greatly improving system throughput. At the same time, the efficient data processing process also shortens the business response time, improves the overall operational efficiency, and brings significant economic benefits to the enterprise. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A flowchart of a ClickHouse-based fusion CDN business data processing method provided in accordance with an embodiment of the present invention.

[0047] Figure 2 A schematic diagram of a specific process of data collection provided by an embodiment of the present invention.

[0048] Figure 3A schematic diagram of a specific flow of data preprocessing provided by an embodiment of the present invention.

[0049] Figure 4 A schematic diagram of a specific process of data storage provided by an embodiment of the present invention.

[0050] Figure 5 A schematic diagram of a specific process of data query provided by an embodiment of the present invention.

[0051] Figure 6 A schematic diagram of the structure of a ClickHouse-based fusion CDN business data processing device provided in one embodiment of the present invention.

[0052] The realization of the purpose, functional features and advantages of the invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0054] The content of the present invention is described in detail below with reference to embodiments.

[0055] Converged CDN refers to the integration of multiple network services and technologies to provide more comprehensive services in a unified architecture. It is an expansion and upgrade of traditional CDN, aiming to better meet the diverse needs of the modern Internet and Internet of Things era through multi-functional integration and intelligent optimization.

[0056] Business data processing is an important process that involves systematically storing and managing various types of data closely related to domain names (such as traffic statistics, number of requests, status codes, etc.). This process is widely used in areas such as user access behavior analysis, customer billing management, and in-depth optimization of network performance. For the data storage system that undertakes this task, it must be able to efficiently respond to and process high-frequency write and query requests for massive amounts of data while ensuring immediate accessibility and strict security of the data. In order to ensure the efficiency and reliability of the business data storage system, a series of technical measures and management strategies need to be adopted.

[0057] Reference Figure 1 The figure is a flow chart of a ClickHouse-based integrated CDN business data processing method provided in one embodiment of the present invention.

[0058] In this embodiment, the method includes:

[0059] S11, acquiring the business data of the corresponding fusion CDN vendor by calling the data interface of each fusion CDN vendor in real time.

[0060] Further, in step S11, the data type of the business data includes bandwidth request data, back-to-source bandwidth request data, status code data, regional bandwidth request data, and TopURL data; the business data of the corresponding fusion CDN vendor is obtained by calling the data interface of each fusion CDN vendor in real time, including:

[0061] S11-1, initialize the system by loading the system configuration file, initialize the acquisition task of each data type, and establish a communication connection with the Kafka cluster;

[0062] S11-2, traversing the fusion CDN vendor list according to the configuration file, traversing the domain name list of each fusion CDN vendor based on the fusion CDN vendor list, and acquiring the domain name data of the target domain name;

[0063] S11-3, retrieve the collection time point information of each target domain name from the MySQL database, obtain the business data of the specified collection time point information by calling the data interface of the corresponding integrated CDN vendor, convert it into JSON format and cache it in the system memory;

[0064] S11-4, batch transfer the business data in JSON format cached in the system memory to the Kafka cluster, and update the next collection time point information of the target domain name in the MySQL database to obtain the next round of business data.

[0065] In this embodiment, the method is implemented based on a data storage system, which includes a data acquisition module, a data preprocessing module, a data storage module, and a data query module. The modules work closely together to ensure the stability of the data link from acquisition to storage. The system acquires business data from various fusion CDN vendors in real time through the data acquisition module. The data acquisition module calls the data interface of each fusion CDN vendor in real time to query business data in real time to ensure the comprehensiveness and timeliness of the collected data. Among them, the data types of business data include bandwidth request data, back-to-source bandwidth request data, status code data, regional bandwidth request data, TopURL data, etc. Figure 2 As shown. Specifically:

[0066] (1) The system loads the core configuration file to complete the system initialization work. According to the content of the configuration file, the system initializes the data collection task of each data type and establishes a communication connection with the Kafka cluster. The system's core configuration file will configure the Kafka cluster information (such as the Kafka cluster address, cluster port, Topic name), MySQL cluster information (address, port), and will record in the configuration file which interface needs to be called in real time from which integrated CDN vendor to query data;

[0067] (2) The system traverses the list of integrated CDN cloud vendors in turn according to the configuration file, and then traverses the list of domain names managed by each vendor. This step aims to obtain the business data of the relevant domain names of each target domain name to ensure the coverage of data collection;

[0068] (3) The system retrieves the collection time point information of each target domain name from the MySQL database to ensure the timeliness and continuity of the data collection task; the system obtains the data of the specified collection time point information by calling the API data interface provided by the integrated CDN manufacturer, and the system converts the returned data into a unified JSON format and temporarily caches it in the system memory for subsequent processing;

[0069] (4) The system transfers the JSON formatted data cached in the memory to the Kafka cluster in batches. After the data is successfully sent, the system updates the next collection time point information of the target domain name in the MySQL database to ensure the smooth progress of subsequent data collection tasks.

[0070] S12, using Spark to preprocess the business data including data format standardization, field expansion, and filtering of invalid data, and packaging the preprocessed business data and sending it to the Kafka cluster.

[0071] Furthermore, in step S12, the use of Spark to preprocess the business data including data format standardization, field expansion, and filtering of invalid data, and packaging the preprocessed business data and sending it to the Kafka cluster includes:

[0072] S12-1, using Spark to set the Kafka cluster address, port and Kafka Topic information, and deploying Spark tasks in a distributed environment and consuming the business data in the Kafka cluster in real time after the Spark tasks are initialized;

[0073] S12-2, traversing the received business data and parsing the business data in JSON format one by one, filtering the business data that fails to be parsed or does not meet the format requirements as invalid data;

[0074] S12-3, cache the filtered business data into the local memory queue. When the amount of data in the local memory queue reaches the data amount threshold, the batch sending mechanism is triggered to batch transmit the current batch of business data to the Kafka cluster.

[0075] In this embodiment, the data preprocessing module is designed based on the Spark framework, and uses its powerful parallel computing capabilities to perform efficient cleaning operations on the collected business data. The cleaning process includes data format standardization, field expansion, filtering of invalid data, etc., to ensure the accuracy and consistency of the data. The cleaned business data is packaged and sent to the Kafka message queue for data distribution and asynchronous processing. This module can consume data in batches to ensure the real-time and high efficiency of large-scale data processing. Figure 3 As shown. Specifically:

[0076] (1) Set the Kafka cluster address, port, and Kafka Topic information of the data acquisition module in the Spark task to ensure that Spark can correctly connect to and obtain the data source; deploy the Spark task in the distributed environment and complete the task initialization to ensure smooth operation in the distributed architecture;

[0077] (2) Real-time consumption of business data from the Kafka cluster;

[0078] (3) The system traverses the received batch business data and parses the JSON format data packets one by one. For data packets that fail to parse or whose format does not meet the standard, the system marks them as dirty data (invalid data) and filters them. At the same time, the system verifies the validity of the business data obtained after filtering. If the data is judged to be invalid, it is also filtered to ensure the accuracy of the cleaned business data.

[0079] (4) For the business data obtained after parsing and validity verification, the system adds it to the local cache queue A. This step is to temporarily store valid business data to facilitate subsequent batch processing and transmission operations;

[0080] (5) When the amount of data in the local cache queue A reaches the set threshold (such as 5,000 records), the system will trigger the batch sending mechanism to transfer the batch of business data to the Kafka cluster in batch form. This step ensures system performance and stability in a large data volume processing environment by optimizing transmission efficiency.

[0081] S13, by creating a distributed data table based on the ReplacingMergeTree engine in the ClickHouse database, using Spark to consume the business data in the Kafka cluster in real time, and writing the business data in batches into the distributed data table of the ClickHouse database.

[0082] Further, in step S13, by creating a distributed data table based on the ReplacingMergeTree engine in the ClickHouse database, using Spark to consume the business data in the Kafka cluster in real time, and writing the business data in batches into the distributed data table of the ClickHouse database, including:

[0083] S13-1, by creating a distributed data table based on the ReplacingMergeTree engine in the ClickHouse database, using Spark to consume the business data in the Kafka cluster in real time and cache it in the local memory queue;

[0084] S13-2, when the data volume of the business data in the local memory queue reaches the data volume threshold, the business data is written in batches to the distributed data table of the ClickHouse database.

[0085] S13-3, using the Kafka Offset update mechanism to automatically update the consumption offset of the Kafka cluster to track the consumption progress of the business data.

[0086] In this embodiment, in the data warehousing module, the system uses the Spark framework to read business data from the Kafka message queue and writes the business data in batches to the ClickHouse database. This process uses Spark's high-performance distributed computing capabilities, and the system can maintain stable performance and efficient processing capabilities in high-concurrency write scenarios. Create a distributed data table based on the ReplacingMergeTree engine in the ClickHouse database to support efficient data writing and automatic replacement of redundant data; use the Spark framework to consume data streams in the Kafka cluster in real time, and write data to ClickHouse in batches after the data batch reaches the set threshold to ensure load balancing and storage efficiency; after the data is successfully written, the system automatically updates Kafka's consumption offset, accurately tracks data processing progress, and prevents data duplication or omissions, thereby improving the continuity and consistency of data processing. Reference Figure 4 As shown. Specifically:

[0087] (1) First, create a distributed data table based on the ReplacingMergeTree engine in the ClickHouse database. The structure of this data table integrates a multi-node data sharding mechanism to support efficient data writing and automatic replacement of redundant data, providing optimized support for subsequent data storage;

[0088] (2) Configure the connection between the Spark cluster and the Kafka cluster. This mainly involves setting the address information, Topic information, and serial number method of the Kafka cluster, so that the Spark task can accurately consume the specified data from the specified Kafka cluster. Then, read the Kafka data stream in the Spark task, parse the JSON format data into formatted structure data, filter invalid or erroneous data, remove duplicates and correct missing fields, and convert and map the data format according to business needs. Finally, write the data to the ClickHouse cluster, and efficiently write the processed data to the ClickHouse data table through batch insertion to ensure that the data matches the ClickHouse data table fields, including data type and field order.

[0089] (3) After the data is successfully written, the system promptly adjusts the consumption offset of the Kafka cluster through an automated Kafka Offset update mechanism. This process ensures that the system can accurately track the consumption progress of business data, avoid duplicate or missing data, and improve the continuity and consistency of data processing.

[0090] S14, retrieve the target business data from the ClickHouse database according to the user's query data request, and convert the target business data into JSON format and return it as a query result.

[0091] Further, in step S14, the target business data is retrieved from the ClickHouse database according to the user's query data request, and the target business data is converted into JSON format and returned as a query result, including:

[0092] S14-1, setting metadata information of distributed data tables in the ClickHouse database through a configuration file, wherein the metadata information includes address information and table information;

[0093] S14-2, initialize the ClickHouse database, and according to the data type queried in the query data request, call the corresponding query logic to retrieve data from the distributed data tables of different ClickHouse databases in turn to obtain the target business data;

[0094] S14-3, using the data rendering mechanism to format the target business data in JSON, and returning the processed query results to the front end.

[0095] In this embodiment, the main function of the data query module is to read data from the ClickHouse database and convert the query results into JSON format and return them to the application. As the core data storage and analysis engine, ClickHouse provides powerful data query and analysis functions. Through its column storage and parallel computing architecture, ClickHouse can efficiently process large-scale data queries and support complex aggregation and analysis operations. Users or applications can interact directly with the ClickHouse database through the data query module to obtain real-time and accurate analysis query results. Figure 5 As shown. Specifically:

[0096] (1) First, configure the metadata information of the ClickHouse database table in the configuration file. The configuration file defines the structure, connection parameters, and address information of the database table, and provides basic configuration information for the data query module;

[0097] (2) After the data query module is started, the system performs the initialization operation of the ClickHouse database connection. By establishing a persistent connection with the ClickHouse database, the system ensures that it can respond quickly when a query data request arrives, thereby achieving efficient data interaction;

[0098] (3) According to the query data type of the user's query request, the system calls the corresponding query logic to retrieve data from different ClickHouse database tables in turn, including: if the user queries the bandwidth request data, then the system will automatically query the data from the ClickHouse bandwidth request data database table; if the user queries the status code data, then the system will automatically query the data from the ClickHouse domain name status code data database table;

[0099] (4) The result set retrieved from the ClickHouse database is formatted in JSON through the designed data rendering mechanism. The system returns the processed business data in JSON format to the front end, providing users with an intuitive query result display to ensure the structured and standardized data transmission. ClickHouse is used to store domain name-related data integrated with CDN. When performing complex multi-dimensional data queries and aggregate analysis, only the data of the relevant columns needs to be read instead of the entire row of data, which significantly reduces the amount of data read, thereby greatly improving the data query performance.

[0100] Reference Figure 6 Shown is a structural diagram of a ClickHouse-based fusion CDN business data processing device provided by an embodiment of the present invention.

[0101] In this embodiment, the device 20 includes:

[0102] The data acquisition unit 21 is used to acquire the service data of the corresponding converged CDN vendor by calling the data interface of each converged CDN vendor in real time;

[0103] A data preprocessing unit 22 is used to use Spark to preprocess the business data, including data format standardization, field expansion, and filtering of invalid data, and to package the preprocessed business data and send it to the Kafka cluster;

[0104] The data storage unit 23 is used to create a distributed data table based on the ReplacingMergeTree engine in the ClickHouse database, consume the business data in the Kafka cluster in real time using Spark, and write the business data in batches into the distributed data table of the ClickHouse database;

[0105] The data query unit 24 is used to retrieve target business data from the ClickHouse database according to the user's query data request, and convert the target business data into JSON format and return it as a query result.

[0106] Each unit module of the device 20 can respectively execute the corresponding steps in the above method embodiment, so each unit module will not be described in detail here. Please refer to the description of the above corresponding steps for details.

[0107] The embodiment of the present invention further provides a ClickHouse-based integrated CDN business data processing device, which includes the ClickHouse-based integrated CDN business data processing device as described above, wherein the ClickHouse-based integrated CDN business data processing device can be used Figure 6 The structure of the embodiment can be executed accordingly. Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects. For details, please refer to the relevant records in the above embodiments, which will not be repeated here.

[0108] The device includes: a mobile phone, a digital camera, a tablet computer, or other devices with a camera function, or a device with an image processing function, or a device with an image display function. The device may include components such as a memory, a processor, an input unit, a display unit, and a power supply.

[0109] Among them, the memory can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as an image playback function, etc.), etc.; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory may also include a memory controller to provide the processor and the input unit with access to the memory.

[0110] The input unit can be used to receive input digital or character or image information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit of this embodiment includes not only a camera, but also a touch-sensitive surface (such as a touch display) and other input devices.

[0111] The display unit can be used to display information input by the user or information provided to the user and various graphical user interfaces of the device, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit may include a display panel, and optionally, the display panel may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface may cover the display panel, and when the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor to determine the type of touch event, and then the processor provides a corresponding visual output on the display panel according to the type of touch event.

[0112] The embodiment of the present invention further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory in the above embodiment; or a computer-readable storage medium that exists independently and is not installed in a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement Figure 1 The fusion CDN business data processing method based on ClickHouse is shown. The computer-readable storage medium may be a read-only memory, a disk or an optical disk, etc.

[0113] The embodiment of the present invention also provides a computer program product, including a computer program / instruction, which is loaded and executed by a processor to implement Figure 1 A ClickHouse-based integrated CDN business data processing method is shown.

[0114] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, equipment embodiment and storage medium embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0115] Furthermore, in this document, the terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element.

[0116] The above description shows and describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the invention, through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not depart from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A method for processing fusion CDN business data based on ClickHouse, characterized in that: The method comprises: By calling the data interfaces of each integrated CDN vendor in real time, the business data of the corresponding integrated CDN vendor can be obtained; Using Spark to preprocess the business data, including data format standardization, field expansion, and filtering of invalid data, and packaging the preprocessed business data and sending it to the Kafka cluster; By creating a distributed data table based on the ReplacingMergeTree engine in the ClickHouse database, using Spark to consume the business data in the Kafka cluster in real time, and writing the business data in batches into the distributed data table of the ClickHouse database; According to the user's query data request, the target business data is retrieved from the ClickHouse database, and the target business data is converted into JSON format and returned as the query result.

2. According to a ClickHouse-based fusion CDN business data processing method according to claim 1, it is characterized in that: The data types of the business data include bandwidth request data, back-to-source bandwidth request data, status code data, regional bandwidth request data, and TopURL data; The process of acquiring the service data of the corresponding fusion CDN vendor by calling the data interface of each fusion CDN vendor in real time includes: Initialize the system by loading the system configuration file, initialize the acquisition task of each data type, and establish a communication connection with the Kafka cluster; Traverse the fusion CDN vendor list according to the configuration file, traverse the domain name list of each fusion CDN vendor based on the fusion CDN vendor list, and obtain the domain name data of the target domain name; Retrieve the collection time point information of each target domain name from the MySQL database, obtain the business data of the specified collection time point information by calling the data interface of the corresponding integrated CDN vendor, convert it into JSON format and cache it in the system memory; The business data in JSON format cached in the system memory is transferred in batches to the Kafka cluster, and the next collection time point information of the target domain name in the MySQL database is updated to obtain the next round of business data.

3. According to a ClickHouse-based fusion CDN business data processing method according to claim 1, it is characterized in that: The using of Spark to preprocess the business data including data format standardization, field expansion, and filtering of invalid data, and packaging the preprocessed business data and sending it to the Kafka cluster, includes: Use Spark to set the Kafka cluster address, port, and Kafka Topic information, and deploy Spark tasks in a distributed environment. After the Spark tasks are initialized, consume the business data in the Kafka cluster in real time. By traversing the received business data and parsing the business data in JSON format one by one, the business data that fails to be parsed or does not meet the requirements in format are filtered as invalid data; The filtered business data is cached in the local memory queue. When the amount of data in the local memory queue reaches the data volume threshold, the current batch of business data is batch-transferred to the Kafka cluster by triggering the batch sending mechanism.

4. According to a ClickHouse-based fusion CDN business data processing method according to claim 1, it is characterized in that: The method creates a distributed data table based on the ReplacingMergeTree engine in the ClickHouse database, uses Spark to consume the business data in the Kafka cluster in real time, and writes the business data in batches to the distributed data table of the ClickHouse database, including: By creating a distributed data table based on the ReplacingMergeTree engine in the ClickHouse database, Spark is used to consume the business data in the Kafka cluster in real time and cache it in the local memory queue; When the amount of business data in the local memory queue reaches the data amount threshold, the business data is written in batches to the distributed data table of the ClickHouse database.

5. According to claim 4, a method for processing integrated CDN business data based on ClickHouse is characterized in that: After the business data is written in batches into the distributed data table of the ClickHouse database, further comprising: The Kafka Offset update mechanism is used to automatically update the consumption offset of the Kafka cluster to track the consumption progress of the business data.

6. According to a ClickHouse-based integrated CDN business data processing method according to claim 1, it is characterized in that: Before retrieving the target business data from the ClickHouse database according to the user's query data request and converting the target business data into JSON format and returning it as a query result, it includes: The metadata information of the distributed data table in the ClickHouse database is set through the configuration file, and the metadata information includes address information and table information.

7. According to claim 6, a method for processing integrated CDN business data based on ClickHouse is characterized in that: The method of retrieving target business data from the ClickHouse database according to the user's query data request and converting the target business data into JSON format as the query result is returned, including: Initialize the ClickHouse database, and according to the data type queried in the query data request, call the corresponding query logic to retrieve data from the distributed data tables of different ClickHouse databases in turn to obtain the target business data; The target business data is formatted in JSON using a data rendering mechanism, and the processed query results are returned to the front end.

8. A fusion CDN business data processing device based on ClickHouse, characterized in that: The device comprises: A data acquisition unit, used to acquire the business data of the corresponding fusion CDN vendor by calling the data interface of each fusion CDN vendor in real time; A data preprocessing unit, used to use Spark to preprocess the business data, including data format standardization, field expansion, and filtering of invalid data, and to package the preprocessed business data and send it to the Kafka cluster; A data storage unit is used to create a distributed data table based on the ReplacingMergeTree engine in the ClickHouse database, consume the business data in the Kafka cluster in real time using Spark, and write the business data in batches into the distributed data table of the ClickHouse database; The data query unit is used to retrieve target business data from the ClickHouse database according to the user's query data request, and convert the target business data into JSON format and return it as a query result.

9. A fusion CDN business data processing device based on ClickHouse, characterized in that: It includes a processor, a memory and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of a ClickHouse-based fusion CDN business data processing method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program / instruction, wherein the computer program / instruction is executed by a processor to implement the steps of a ClickHouse-based fusion CDN business data processing method as described in any one of claims 1 to 7.

Images