CDN log statistics method, device and electronic equipment

Abstract

The present invention discloses a CDN log statistics method, device and electronic device. A storage module storing CDN log data is communicatively connected to multiple clusters. The method is applied to a processing device corresponding to any cluster in the multiple clusters, comprising: reading CDN log data from a Hadoop distributed file system in the corresponding cluster into a corresponding Spark partition, using a dimension combination composed of some fields of the CDN log data as a basis for CDN log classification, using the CDN log type as a key and the CDN log numerical dimension of the corresponding type as a value; traversing all corresponding values ​​of each key in the same Spark partition to construct a corresponding mini-heap data structure; merging data in the mini-heap data structures corresponding to the same key in different Spark partitions in the cluster; and synchronizing the data in the mini-heap data structures corresponding to each key obtained after the merger to other clusters.

Description

Technical Field

[0001] The present invention relates to the technical field of big data analysis, and in particular to a CDN log statistics method, device and electronic equipment. Background Art

[0002] With the development of information technology, the rapid growth of network traffic has become a current reality and a future trend. As a technical service that improves network service quality and reduces the pressure on content origins, the use of content delivery networks (CDNs) has also seen explosive growth. When using CDN services, every network request generates a CDN log. To meet data analysis and business-related needs, CDN vendors often need to regularly sort and count the numerical data corresponding to a certain dimension or combination of dimensions in the massive CDN logs, calculating the top N ranked data points for business needs. In scenarios with massive data volumes, using a single machine to process data often leads to performance bottlenecks, making it impossible to meet real-time requirements. Furthermore, with single-machine processing, the reliability of computing and storage services is often unreliable. Network and hardware failures can easily lead to online incidents, causing service unavailability and severely impacting the user experience. Summary of the Invention

[0003] Therefore, the technical problem to be solved by the present invention is to overcome the defects of the existing single machine in processing massive data, such as poor real-time performance and inability to guarantee the reliability of computing and storage services, thereby providing a CDN log statistics method, device and electronic device.

[0004] According to a first aspect, an embodiment of the present invention discloses a CDN log statistics method, wherein a storage module storing CDN log data is communicatively connected to multiple clusters, and the multiple clusters store corresponding CDN log data according to a preset ratio. The method is applied to a processing device corresponding to any cluster in the multiple clusters, including: reading CDN log data from a Hadoop distributed file system in the corresponding cluster into a corresponding partition of Spark, and using a dimension combination composed of some fields of the CDN log data as a basis for CDN log classification, using the CDN log type as a key, and using the CDN log numerical dimension of the corresponding type as a value; traversing all corresponding values ​​of each key in the same Spark partition to construct a corresponding mini-heap data structure; merging data in the mini-heap data structures corresponding to the same key in different Spark partitions in the cluster; and synchronizing the data in the mini-heap data structures corresponding to each key obtained after the merger to other clusters.

[0005] Optionally, the method further includes: obtaining synchronization data in a mini-heap data structure corresponding to each key obtained after the merger in other clusters; and saving the synchronization data in ClickHouse.

[0006] Optionally, traversing all corresponding values ​​of each key in the same Spark partition to construct a corresponding mini-heap data structure includes: traversing the first N values ​​corresponding to each key in the same Spark partition to obtain a mini-heap of size N; traversing other values ​​except the first N values ​​according to the obtained mini-heap of size N, and when the other values ​​are greater than the top data of the mini-heap, replacing the top data in the mini-heap, and reconstructing a mini-heap of size N based on the replaced data.

[0007] Optionally, the merging of the mini-heaps corresponding to the same key in different Spark partitions in the cluster includes: obtaining all values ​​in the mini-heap data structure corresponding to the same key in different Spark partitions, and obtaining a set of corresponding values; traversing the first N values ​​of all values ​​in the set of values ​​according to the same key in different Spark partitions to obtain a mini-heap of corresponding size N; traversing other values ​​in the set of corresponding values ​​except the first N values ​​according to the obtained mini-heap of corresponding size N, and when the other values ​​are greater than the top data of the mini-heap, replacing the top data in the mini-heap, and reconstructing a mini-heap of size N based on the replaced data.

[0008] Optionally, after saving the synchronized data in ClickHouse, the method further includes: when receiving a query data request from the client, responding to the query data request operation of the client and obtaining corresponding data from ClickHouse and feeding it back to the client.

[0009] According to the second aspect, an embodiment of the present invention further discloses a CDN log statistics device, wherein a storage module storing CDN log data is communicatively connected to multiple clusters, and the multiple clusters store corresponding CDN log data according to a preset ratio. The method is applied to a processing device corresponding to any cluster in the multiple clusters, comprising: a reading module for reading CDN log data from the Hadoop distributed file system in the corresponding cluster into a corresponding partition of Spark, and using a dimension combination composed of some fields of the CDN log data as a basis for CDN log classification, using the CDN log type as a key, and using the CDN log numerical dimension of the corresponding type as a value; a construction module for traversing all corresponding values ​​of each key in the same Spark partition to construct a corresponding mini-heap data structure; a merging module for merging data in the mini-heap data structures corresponding to the same key in different Spark partitions in the cluster; and synchronizing the data in the mini-heap data structures corresponding to each key obtained after the merger to other clusters.

[0010] Optionally, the device further includes: a synchronization module for obtaining synchronization data in a mini-heap data structure corresponding to each key obtained after the merger in other clusters; and a storage module for saving the synchronization data in ClickHouse.

[0011] Optionally, the construction module also includes: a first determination module, used to traverse the first N values ​​corresponding to each key in the same Spark partition to obtain a mini-heap of size N; a reconstruction module, used to traverse other values ​​except the first N values ​​according to the obtained mini-heap of size N, and when the other values ​​are greater than the top data of the mini-heap, replace the top data in the mini-heap, and reconstruct a mini-heap of size N based on the replaced data.

[0012] According to a third aspect, an embodiment of the present invention further discloses an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the CDN log statistics method as described in the first aspect or any optional embodiment of the first aspect.

[0013] According to a fourth aspect, an embodiment of the present invention further discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the CDN log statistics method as described in the first aspect or any optional embodiment of the first aspect are implemented.

[0014] The technical solution of the present invention has the following advantages:

[0015] The CDN log statistics method / device provided by the present invention has a storage module storing CDN log data and is communicatively connected to multiple clusters, wherein the multiple clusters store corresponding CDN log data according to a preset ratio. The method is applied to a processing device corresponding to any cluster in the multiple clusters, comprising: reading CDN log data from a Hadoop distributed file system in the corresponding cluster into a corresponding Spark partition, and using a dimension combination composed of some fields of the CDN log data as a basis for CDN log classification, using the CDN log type as a key and the CDN log numerical dimension of the corresponding type as a value; traversing all corresponding values ​​of each key in the same Spark partition to construct a corresponding mini-heap data structure; merging the data in the mini-heap data structure corresponding to the same key in different Spark partitions in the cluster; and synchronizing the data in the mini-heap data structure corresponding to each key obtained after the merger to other clusters. The method of the present invention adopts a distributed architecture to perform sorting calculations and save results according to a certain dimension or a group of dimensions. The data processing capacity can be allocated in any proportion between the primary and backup dual clusters, and both clusters save the full processing results, greatly improving system reliability. At the same time, this method reduces the number of data shuffling times in a distributed environment and improves calculation efficiency by using a mini-heap data structure in the sorting calculation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 This is a flowchart of a specific example of a CDN log statistics method according to an embodiment of the present invention;

[0018] Figure 2 This is a flowchart of a specific example of a CDN log statistics method according to an embodiment of the present invention;

[0019] Figure 3 This is a functional block diagram of a specific example of a CDN log statistics device according to an embodiment of the present invention;

[0020] Figure 4 FIG. 4 is a diagram showing a specific example of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0022] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0023] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; internal connections between two components; wireless connections or wired connections. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0024] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0025] The embodiment of the present invention discloses a CDN log statistics method, wherein a storage module storing CDN log data is connected to multiple clusters for communication, and the multiple clusters store corresponding CDN log data according to a preset ratio. The method is applied to a processing device corresponding to any cluster in the multiple clusters. In the embodiment of the present application, Figure 2As shown, taking the multiple clusters as the main and backup clusters as an example, each cluster contains a log reading and analysis module, a data synchronization module, and a data storage module. The log reading and analysis module is designed based on the open source big data processing framework Spark. Its function is to read the original CDN data log, sort it according to a certain dimension or a group of dimensions and its corresponding data values, and write the calculation results to the data synchronization module; the data synchronization module uses the distributed message queue Kafka, and establishes corresponding Kafka topics on the two clusters to receive the calculation results written by the log reading and analysis module deployed on the same cluster, and synchronize the data with each other through Kafka Mirror (a plug-in for Kafka data replication between clusters), and then write the data to the data storage module; the data storage module uses the columnar database management system ClickHouse, and imports and saves the full amount of data from the data synchronization module through ClickHouse sinker (a plug-in for writing Kafka data to ClickHouse), and provides query functions; the main and backup dual clusters process and store the corresponding CDN log data according to the preset ratio, and the preset ratio can be any ratio. As shown Figure 1 As shown, the method includes the following steps:

[0026] Step 101: read CDN log data from the Hadoop distributed file system in the corresponding cluster into the corresponding partition of Spark, and use some fields of the CDN log data to form a dimension combination as the basis for CDN log classification, using the CDN log type as the key and the corresponding type of CDN log numerical dimension as the value.

[0027] Exemplarily, CDN log data is stored on the Hadoop distributed file system of each cluster. The required indicator dimension is selected in the corresponding cluster, and Spark is used to read the data into two types of entities. One is the entity corresponding to the classification dimension group, and the other is the entity composed of the indicator dimension and the data value dimension corresponding to the indicator dimension. The first entity is used as the key, and the second entity is used as the value. One key corresponds to multiple values; the classification dimension combination can be one or more fields in the CDN log data. Specifically, the domain name plus the product type, the two dimensions added together is a dimension combination. There are multiple dimensions in the log data. It can be determined according to the research object which dimensions are combined together as a classification mark.

[0028] Step 102: For each key in the same Spark partition, all corresponding values ​​are traversed to construct a corresponding mini-heap data structure. For example, for each key in the same Spark partition, all corresponding values ​​of the Spark partition are traversed to construct a corresponding mini-heap data structure.

[0029] Step 103: Merge the data in the mini-heap data structure corresponding to the same key in different Spark partitions in the cluster. For example, the mini-heap data corresponding to the same key in different partitions within the same cluster are merged to obtain the merged corresponding mini-heap data, and the data in the mini-heap is the data statistics result under the grouping dimension.

[0030] Step 104: Synchronize the data in the small top heap data structure corresponding to each key obtained after the merger to other clusters. Exemplarily, the data in the small top heap data structure corresponding to each key obtained after the merger is transmitted to other clusters via Kafka Mirror.

[0031] Specifically, each cluster consists of 30 servers, each server has 2 32-core CPUs, 256GB of memory, 6 6T mechanical hard drives, and a 10G network card. Each CDN log contains dimensional information such as product number, domain name, customer IP, and traffic value, and calculates the top 200 customer IPs with the highest traffic value for each domain name of each product; the deployment of Spark, Kafka, ClickHouse and other components and their dependent components and environments are completed on two distributed clusters, and the CDN log data is divided into approximately 50%:50% and stored on the HDFS of the two clusters respectively, and the CDN logs with the same domain name are guaranteed to be stored in the same cluster; the log reading and analysis module sums and accumulates the traffic value with product number, domain name, and customer IP as the aggregation group, and then uses the dimension combination of product number and domain name as the CDN log type division mark to form a key entity, and combines the customer IP and the summed traffic value into a value entity. One key entity corresponds to multiple value entities; 45 Spark Executors are set in each cluster, and the Executor memory is set to 16G. The cores are set to 6, and each Kafka topic is set to 10 partitions. If the CDN log contains 1,000 domain names and 6 product numbers, theoretically 1.2 million records will be output and saved. However, due to data skew, the actual data output is less than the theoretical value. With the above configuration, the entire method takes approximately 4 to 10 minutes to read and calculate data for 5 billion CDN logs, meeting the efficiency requirements for ranking calculation in large-scale data scenarios.

[0032] The CDN log statistics method provided by the present invention comprises a storage module storing CDN log data and communicating with multiple clusters, wherein the multiple clusters store corresponding CDN log data according to a preset ratio, and the method is applied to a processing device corresponding to any cluster in the multiple clusters, comprising: reading CDN log data from a Hadoop distributed file system in the corresponding cluster into a corresponding partition of Spark, and using a dimension combination composed of some fields of the CDN log data as a basis for CDN log type classification, using the CDN log type as a key, and using the CDN log numerical dimension of the corresponding type as a value; traversing all corresponding values ​​of each key in the same Spark partition to construct a corresponding mini-heap data structure; merging the data in the mini-heap data structure corresponding to the same key in different Spark partitions in the cluster; and synchronizing the data in the mini-heap data structure corresponding to each key obtained after the merger to other clusters. The method of the present invention adopts a distributed architecture to perform sorting calculations and save results according to a certain dimension or a group of dimensions, and can allocate data processing capacity in any proportion between the primary and backup dual clusters, and both clusters save the full processing results, thereby improving system reliability. At the same time, this method reduces the number of data shuffling times in a distributed environment and improves calculation efficiency by using a small top heap data structure in the sorting calculation process.

[0033] As an optional embodiment of the present invention, the method further includes: obtaining synchronization data in a mini-heap data structure corresponding to each key obtained after the merger in other clusters; and saving the synchronization data in ClickHouse.

[0034] Exemplarily, using the columnar database management system ClickHouse, the full amount of data in the mini-heap data structure corresponding to each key in this cluster and synchronized from other clusters is obtained from the Kafka topic of this cluster through ClickHouse sinker and written into ClickHouse.

[0035] As an optional embodiment of the present invention, the traversal of all corresponding values ​​of each key in the same Spark partition to construct a corresponding mini-heap data structure includes: traversing the first N values ​​corresponding to each key in the same Spark partition to obtain a mini-heap of size N; traversing other values ​​except the first N values ​​according to the obtained mini-heap of size N, when the other values ​​are greater than the top data of the mini-heap, replacing the top data in the mini-heap, and reconstructing a mini-heap of size N based on the replaced data.

[0036] For example, within the same partition, for each key, the first N values ​​are traversed to establish a mini-heap data structure of size N, and then the remaining values ​​are traversed. If the value is larger than the top data of the mini-heap, it is replaced and the replaced data structure is readjusted to the mini-heap. The readjusted mini-heap is a mini-heap of size N.

[0037] As an optional embodiment of the present invention, the merging of the mini-heaps corresponding to the same key in different Spark partitions in the cluster includes: obtaining all values ​​in the mini-heap data structure corresponding to the same key in different Spark partitions, and obtaining a set of corresponding values; traversing the first N values ​​of all values ​​in the set of values ​​according to the same key in different Spark partitions to obtain a mini-heap of corresponding size N; traversing other values ​​in the corresponding set of values ​​except the first N values ​​according to the obtained mini-heap of corresponding size N, and when the other values ​​are greater than the top data of the mini-heap, replacing the top data in the mini-heap, and reconstructing a mini-heap of size N based on the replaced data.

[0038] For example, the mini-heaps corresponding to the same key in different partitions are merged. The merging method is to extract all data values ​​corresponding to the same key in different partitions as a set. According to the previous mini-heap creation and adjustment method, the set data is traversed to calculate the final mini-heap. After the merger, each key corresponds to a mini-heap of size N, and the data in the mini-heap is the top N data under the grouping dimension.

[0039] As an optional embodiment of the present invention, after saving the synchronized data in ClickHouse, the method further includes: when receiving a query data request from the client, responding to the query data request operation of the client and obtaining corresponding data from ClickHouse and feeding it back to the client.

[0040] For example, when a request for querying data is received from a client, ClickHouse responds to the client's query data request operation, and at the same time obtains the corresponding data from ClickHouse and feeds it back to the client.

[0041] The embodiment of the present invention also discloses a CDN log statistics device, wherein a storage module storing CDN log data is connected to multiple clusters for communication, and the multiple clusters store corresponding CDN log data according to a preset ratio. The method is applied to a processing device corresponding to any cluster in the multiple clusters, such as Figure 3As shown, the device includes: a reading module 201, which reads CDN log data from the Hadoop distributed file system in the corresponding cluster into the corresponding partition of Spark, and uses a dimension combination composed of some fields of the CDN log data as the basis for CDN log classification, uses the CDN log type as the key, and uses the numerical dimension of the CDN log of the corresponding type as the value; a construction module 202, which is used to traverse all the corresponding values ​​of each key in the same Spark partition to construct a corresponding mini-heap data structure; a merging module 203, which is used to merge the data in the mini-heap data structure corresponding to the same key in different Spark partitions in the cluster; and a synchronization module 204, which is used to synchronize the data in the mini-heap data structure corresponding to each key obtained after the merger to other clusters.

[0042] The CDN log statistics device provided by the present invention has a storage module storing CDN log data and is connected to multiple clusters for communication. The multiple clusters store corresponding CDN log data according to a preset ratio. The method is applied to the processing equipment corresponding to any cluster in the multiple clusters. The device includes: a reading module, which reads CDN log data from the Hadoop distributed file system in the corresponding cluster into the corresponding partition of Spark, and uses a dimension combination composed of some fields of the CDN log data as a CDN log classification basis, uses the CDN log type as a key, and uses the CDN log numerical dimension of the corresponding type as a value; a construction module, which is used to traverse all corresponding values ​​of each key in the same Spark partition to construct a corresponding mini-heap data structure; a merging module, which is used to merge the data in the mini-heap data structure corresponding to the same key in different Spark partitions in the cluster; and a synchronization module, which is used to synchronize the data in the mini-heap data structure corresponding to each key obtained after the merger to other clusters. The device of the present invention adopts a distributed architecture to perform sorting calculations and save results according to a certain dimension or a group of dimensions. The data processing capacity can be allocated in any proportion in the primary and backup dual clusters, and both clusters save the full processing results, greatly improving the system reliability. At the same time, the device reduces the number of data shuffling times in a distributed environment and improves computing efficiency by using a small top heap data structure in the sorting calculation process.

[0043] As an optional embodiment of the present invention, the device also includes: an acquisition module for acquiring the synchronization data in the mini-heap data structure corresponding to each key obtained after the merger in other clusters; and a storage module for saving the synchronization data in ClickHouse.

[0044] As an optional embodiment of the present invention, the construction module also includes: a first determination module, used to traverse the corresponding first N values ​​of each key in the same Spark partition to obtain a mini-heap of size N; a reconstruction module, used to traverse other values ​​except the first N values ​​according to the obtained mini-heap of size N, and when the other values ​​are greater than the top data of the mini-heap, replace the top data in the mini-heap, and reconstruct a mini-heap of size N based on the replaced data.

[0045] As an optional embodiment of the present invention, the merging module also includes: a second determination module, used to obtain all values ​​in the mini-heap data structure corresponding to the same key in different Spark partitions, and obtain a set of corresponding values; a third determination module: used to traverse the first N values ​​of all values ​​in the set of values ​​according to the same key in different Spark partitions to obtain a mini-heap of corresponding size N; a merging sub-module, used to traverse the other values ​​in the corresponding set of values ​​except the first N values ​​according to the obtained mini-heap of corresponding size N, and when the other values ​​are greater than the top data of the mini-heap, replace the top data in the mini-heap, and reconstruct a mini-heap of size N based on the replaced data.

[0046] As an optional embodiment of the present invention, the storage module further includes: a response module for responding to the client's query data request operation and obtaining corresponding data from ClickHouse and feeding it back to the client when the client's query data request is obtained.

[0047] The embodiment of the present invention further provides an electronic device, such as Figure 4 As shown, the electronic device may include a processor 401 and a memory 402, wherein the processor 401 and the memory 402 may be connected via a bus or other means. Figure 4 The bus connection is taken as an example.

[0048] The processor 401 may be a central processing unit (CPU). The processor 401 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0049] Memory 402, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as the program instructions / modules corresponding to the CDN log statistics method in the embodiments of the present invention. Processor 401 executes the non-transitory software programs, instructions, and modules stored in memory 402 to perform various processor functions and data processing, thereby implementing the CDN log statistics method in the above-mentioned method embodiments.

[0050] The memory 402 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created by the processor 401, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 402 may optionally include a memory remotely located relative to the processor 401, and these remote memories may be connected to the processor 401 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0051] The one or more modules are stored in the memory 402 and when executed by the processor 401, perform the following steps: Figure 1 The CDN log statistics method in the illustrated embodiment.

[0052] For details of the above electronic equipment, please refer to Figure 1 The corresponding descriptions and effects in the embodiments shown can be understood and will not be repeated here.

[0053] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD). The storage medium can also include a combination of the above-mentioned types of memory.

[0054] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A CDN log statistics method, characterized in that: The storage module storing CDN log data is in communication with multiple clusters, which store corresponding CDN log data in a preset ratio. Each cluster includes a log reading and analysis module, a data synchronization module, and a data storage module. The log reading and analysis module is used to read the original CDN data log, sort it according to the dimension and its corresponding data value, and write the calculation results to the data synchronization module. The data synchronization module is used to receive the calculation results written by the log reading and analysis module deployed on the same cluster, synchronize data with other clusters through Kafka Mirror, and write the synchronized data to the data storage module. The method is applied to the processing device corresponding to any cluster in multiple clusters, including: Read CDN log data from the Hadoop distributed file system in the corresponding cluster into the corresponding partition of Spark. Use some fields of the CDN log data to form a dimension combination as the basis for CDN log classification, using the CDN log type as the key and the corresponding type of CDN log numerical dimension as the value. For each key in the same Spark partition, traverse all corresponding values ​​and build the corresponding mini-heap data structure; Merging data in the mini-heap data structure corresponding to the same key in different Spark partitions in the cluster; Synchronize the data in the small top heap data structure corresponding to each key obtained after the merger to other clusters; The process of traversing all corresponding values ​​of each key in the same Spark partition and constructing a corresponding mini-heap data structure includes: For each key in the same Spark partition, traverse the corresponding first N values ​​to obtain a small top heap of size N; According to the obtained mini-top heap of size N, other values ​​except the first N values ​​are traversed. When the other values ​​are greater than the top data of the mini-top heap, the top data in the mini-top heap is replaced, and a mini-top heap of size N is reconstructed based on the replaced data.

2. The method according to claim 1, characterized in that The method further comprises: Obtain the synchronized data in the mini-heap data structure corresponding to each key obtained after the merger in other clusters; Save the synchronized data in ClickHouse.

3. The method according to claim 1, characterized in that The merging of the mini-heaps corresponding to the same key in different Spark partitions in the cluster includes: Get all values ​​in the mini-heap data structure corresponding to the same key in different Spark partitions, and get the corresponding value set; Traverse the first N values ​​of all values ​​in the set of values ​​according to the same key in different Spark partitions to obtain a small top heap of corresponding size N; According to the obtained mini-heap of corresponding size N, other values ​​except the first N values ​​in the corresponding value set are traversed. When the other values ​​are greater than the top data of the mini-heap, the top data in the mini-heap is replaced, and a mini-heap of size N is reconstructed based on the replaced data.

4. The method according to claim 2, characterized in that After saving the synchronized data in ClickHouse, the method further includes: When receiving the client's query data request, respond to the client's query data request operation and obtain the corresponding data from ClickHouse and feed it back to the client.

5. A CDN log statistics device, characterized in that: The storage module storing CDN log data is in communication with multiple clusters, which store corresponding CDN log data in a preset ratio. Each cluster includes a log reading and analysis module, a data synchronization module, and a data storage module. The log reading and analysis module is used to read the original CDN data log, sort it according to the dimension and its corresponding data value, and write the calculation results to the data synchronization module. The data synchronization module is used to receive the calculation results written by the log reading and analysis module deployed on the same cluster, synchronize data with other clusters through Kafka Mirror, and write the synchronized data to the data storage module. The device is applied to the processing equipment corresponding to any cluster in the multiple clusters, including: The reading module is used to read CDN log data from the Hadoop distributed file system in the corresponding cluster into the corresponding partition of Spark, and use the dimension combination composed of some fields of the CDN log data as the basis for CDN log classification, with the CDN log type as the key and the corresponding type of CDN log numerical dimension as the value; The construction module is used to traverse all the corresponding values ​​of each key in the same Spark partition and build the corresponding mini-heap data structure; A merging module, configured to merge data in the mini-heap data structure corresponding to the same key in different Spark partitions in the cluster; The synchronization module is used to synchronize the data in the small top heap data structure corresponding to each key obtained after the merger to other clusters; The building blocks also include: The first determination module is used to traverse the first N values ​​corresponding to each key in the same Spark partition to obtain a small top heap of size N; A reconstruction module is used to traverse other values ​​except the first N values ​​according to the obtained mini-top heap of size N, and when the other values ​​are greater than the top data of the mini-top heap, replace the top data in the mini-top heap, and reconstruct a mini-top heap of size N according to the replaced data.

6. The device according to claim 5, characterized in that The device further comprises: An acquisition module is used to obtain the synchronization data in the mini-heap data structure corresponding to each key obtained after the merger in other clusters; The storage module is used to save the synchronization data in ClickHouse.

7. An electronic device, characterized in that: include: at least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the CDN log statistics method as described in any one of claims 1-4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the CDN log statistics method according to any one of claims 1 to 4 are implemented.

Images