Big data real-time statistical analysis method and system based on distributed computing architecture

By using the combination of data sharding, Hyper-Log-Log and Count-Min-Sketch algorithms in distributed computing architecture, the calculation complexity and resource bottleneck problems of traditional statistical methods in big data processing are solved, and efficient Real-time statistical analysis and data management.

CN118760714BActive Publication Date: 2024-11-05NORTHEASTERN UNIV AT QINHUANGDAO
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411241796.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-05
Publication Date
2024-11-05
Estimated Expiration
2044-09-05

AI Technical Summary

Technical Problem

Traditional statistical methods have high computational complexity when processing large data sets, making it difficult to achieve real-time processing, especially when frequently updated and queried, and are prone to resource bottlenecks in distributed systems and difficult to adapt to data diversity.

Method used

Using a big data real-time statistical analysis method based on a distributed computing architecture, the data is distributed to different nodes through data sharding strategy for parallel processing, and the approximate statistics are performed by combining Hyper-Log-Log and Count-Min-Sketch algorithms, and the distribution is used. Coordinate the results in aggregation of results and display the sorting results in real time through Grafana tool.

Benefits of technology

It improves data processing efficiency, ensures data processing effect, simplifies resource management, adapts to data diversity, and improves enterprise management efficiency of data sets.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118760714B_ABST
    Figure CN118760714B_ABST
Patent Text Reader

Abstract

The present invention discloses a real-time statistical analysis method and system for big data based on a distributed computing architecture, which relates to the technical field of big data statistics. The data in a data set is distributed to different computing nodes based on a data sharding strategy, a structural score is generated for the data set, and based on the comparison result between the structural score and the scoring threshold, it is judged whether it is necessary to perform approximate statistics on the data set. When it is judged that approximate statistics are needed, on each computing node, the number of independent elements in the data stream is estimated by the Hyper‑Log‑Log algorithm, and the frequency of each element in the data stream is estimated by the Count‑Min‑Sketch algorithm, and then the data is deduplicated. The analysis system distributes the collected data to different nodes for parallel processing, and then judges whether it is necessary to perform approximate statistics on the data in combination with the analysis algorithm, which ensures the data processing effect while helping to improve the data processing efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of big data statistics technology, and in particular to a big data real-time statistical analysis method and system based on a distributed computing architecture. Background Art

[0002] Big data statistical analysis is a technology based on the collection, storage, processing and analysis of large amounts of data. It aims to extract valuable insights and information from massive and diverse data sets. With the rapid development of information technology and data generation, enterprises, governments and research institutions are increasingly relying on big data analysis to support decision making, optimize business processes and enhance competitiveness.

[0003] The prior art has the following deficiencies:

[0004] Traditional statistical methods have high computational complexity when processing large data sets and are difficult to achieve real-time processing, especially when frequent updates and queries are required. The efficiency is low. When processing massive amounts of data, accurate statistics usually require a large amount of memory and storage resources, which is particularly evident in distributed systems, which can easily lead to resource bottlenecks and make it difficult to adapt to the diversity of data.

[0005] Based on this, the present invention proposes a real-time statistical analysis method and system for big data based on a distributed computing architecture. The collected data is distributed to different nodes for parallel processing, and then the analysis algorithm is combined to determine whether approximate statistics of the data are needed. This ensures the data processing effect while helping to improve data processing efficiency. Summary of the invention

[0006] The purpose of the present invention is to provide a big data real-time statistical analysis method and system based on a distributed computing architecture to address the deficiencies in the background technology.

[0007] In order to achieve the above object, the present invention provides the following technical solution: a big data real-time statistical analysis method based on a distributed computing architecture, the analysis method comprising the following steps:

[0008] Use message queue tools to collect and preprocess data streams to create data sets, and distribute data in the data sets to different computing nodes based on data sharding strategies;

[0009] After obtaining the data volume, data integrity, and matching similarity of the data set, a structural score is generated for the data set. Based on the comparison result between the structural score and the score threshold, it is determined whether approximate statistics of the data set are required. When it is determined that approximate statistics are required, the number of independent elements in the data stream is estimated by the Hyper-Log-Log algorithm on each computing node, and the frequency of each element in the data stream is estimated by the Count-Min-Sketch algorithm, and then the data is deduplicated.

[0010] After each computing node completes processing, all data are aggregated using the distributed coordination service to obtain global statistical results. The deduplication processing results and the structural score are combined to generate a quality score for the data set. After all data sets are sorted according to the quality score, the sorting results are displayed in real time through the Grafana tool.

[0011] In a preferred embodiment, after obtaining the data volume, data integrity and matching similarity of the data set, generating a structure score for the data set includes the following steps:

[0012] The data volume, data integrity and matching similarity are calculated comprehensively to obtain the structure score, which is expressed as: , where Score the structure. is the amount of data, For data integrity, To match similarity, , , are the proportional coefficients of data volume, data integrity and matching similarity, respectively, and , , Both are greater than 0.

[0013] In a preferred embodiment, based on the comparison result between the structure score and the score threshold, determining whether it is necessary to perform approximate statistics on the data set includes the following steps:

[0014] The obtained structural score is compared with the preset scoring threshold. The scoring threshold is used to determine whether it is necessary to perform approximate statistics on the data set. If the structural score is less than the scoring threshold, it is determined that it is not necessary to perform approximate statistics on the data set. If the structural score is greater than or equal to the scoring threshold, it is determined that it is necessary to perform approximate statistics on the data set.

[0015] In a preferred embodiment, on each computing node, the number of independent elements in the data stream is estimated by the Hyper-Log-Log algorithm, and the frequency of each element in the data stream is estimated by the Count-Min-Sketch algorithm, and then the data is deduplicated, including the following steps:

[0016] Calculate the hash value for each element in the data stream, update the Hyper-Log-Log register according to the hash value, use the number of leading zeros of the hash value to update the corresponding register, and calculate the estimated value of the number of independent elements. The expression is: , where is an estimated value, is the number of registers, is a constant used to calibrate the registers, It is The value of a register;

[0017] For each element in the data stream, use each hash function to calculate the hash value and increase 1 in the corresponding counter. For the queried element, use all hash functions to calculate the hash value, obtain the estimated frequency from the corresponding counter, and take the minimum value as the frequency estimate;

[0018] Collect Hyper-Log-Log structures from all computing nodes and merge them to obtain a global estimate of the number of independent elements. Collect Count-Min-Sketch structures from each computing node and merge them to obtain a global estimate of element frequency. Based on the frequency estimate of Count-Min-Sketch, determine whether there is duplicate data and delete duplicate data whose frequency estimate is greater than the frequency threshold.

[0019] In a preferred embodiment, the deduplication processing result and the structure score are combined to generate a quality score for the data set, and all data sets are sorted according to the quality score, including the following steps:

[0020] Get the number of deduplication processing times of the dataset on each computing node, sum the number of deduplication processing times on all computing nodes to get the total number of deduplication processing times, normalize the total number of deduplication processing times and the structure score, map the value range of the total number of deduplication processing times and the structure score to [0,1], get the deduplication normalized value and the structure score normalized value, subtract the deduplication normalized value from the structure score normalized value to get the quality score, and sort all datasets from large to small according to the quality score.

[0021] In a preferred embodiment, the calculation logic of the matching similarity is: first calculate the Jaccard similarity coefficient between the collected data set and the existing data set in the database, , where is the Jaccard similarity coefficient, Indicates the number of elements in the intersection of the collected dataset and the existing dataset in the database. It represents the number of elements in the union of the collected dataset and the existing dataset in the database, and then selects the maximum Jaccard similarity coefficient as the matching similarity. The expression is: , where To match similarity, Indicates the collection of data sets and database Jaccard similarity coefficient of the data set, Indicates taking the maximum value.

[0022] In a preferred embodiment, distributing data in a data set to different computing nodes based on a data sharding strategy includes the following steps:

[0023] Define sharding rules based on the sharding strategy and configure the rules in the analysis system. The rules are used to guide how to allocate data to different shards.

[0024] The data stream is transmitted to each computing node in the distributed system through the message queue tool. The data received by each node is the shard data corresponding to the node.

[0025] When the amount of data or query load changes, dynamically increase or decrease the number of shards, and migrate data when adjusting the sharding strategy or expanding the shards.

[0026] A big data real-time statistical analysis system based on a distributed computing architecture, including a data set establishment module, a data distribution module, a data set analysis module, a data deduplication module, and an aggregation display module:

[0027] Data set building module: Use message queue tools to collect and pre-process data streams to build data sets;

[0028] Data distribution module: distributes data in the data set to different computing nodes based on the data sharding strategy;

[0029] Dataset analysis module: After obtaining the data volume, data integrity and matching similarity of the data set, a structural score is generated for the data set. Based on the comparison between the structural score and the score threshold, it is determined whether approximate statistics of the data set are required;

[0030] Data deduplication module: When approximate statistics are required, the Hyper-Log-Log algorithm is used on each computing node to estimate the number of independent elements in the data stream, and the Count-Min-Sketch algorithm is used to estimate the frequency of each element in the data stream and then perform data deduplication processing;

[0031] Aggregation display module: After each computing node completes processing, all data are aggregated using the distributed coordination service to obtain global statistical results. The deduplication processing results and the structural score are combined to generate a quality score for the data set. After all data sets are sorted according to the quality score, the sorting results are displayed in real time through the Grafana tool.

[0032] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0033] 1. Based on the data sharding strategy, the present invention distributes the data in the data set to different computing nodes according to key, time, and source, obtains the data volume, data integrity, and matching similarity of the data set, and generates a structural score for the data set. Based on the comparison result of the structural score and the score threshold, it is judged whether it is necessary to perform approximate statistics on the data set. When it is judged that approximate statistics are needed, on each computing node, the number of independent elements in the data stream is estimated by the Hyper-Log-Log algorithm, and then the frequency of each element in the data stream is estimated by the Count-Min-Sketch algorithm to deduplicate the data. The analysis system distributes the collected data to different nodes for parallel processing, and then combines the analysis algorithm to determine whether it is necessary to perform approximate statistics on the data, which ensures the data processing effect while helping to improve the data processing efficiency.

[0034] 2. After each computing node completes the processing, the present invention aggregates all data using distributed coordination services to obtain global statistical results, combines the deduplication processing results with the structural score to generate a quality score for the data set, sorts all data sets according to the quality score, and displays the sorting results in real time through the Grafana tool. This makes it easier for enterprises to manage data sets and improves management efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0036] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] Example 1: Please refer to Figure 1 As shown, the big data real-time statistical analysis method based on the distributed computing architecture described in this embodiment includes the following steps:

[0039] After using Apache-Kafka, RabbitMQ and other message queue tools to collect and preprocess data streams, a data set is established. Preprocessing includes data cleaning, format conversion and filtering to ensure data quality and consistency. Based on the data sharding strategy, the data in the data set is distributed to different computing nodes according to key, time, source, etc. After obtaining the data volume, data integrity and matching similarity of the data set, a structural score is generated for the data set. Based on the comparison results between the structural score and the score threshold, it is determined whether approximate statistics of the data set are required. When it is determined that approximate statistics are required, on each computing node, the Hyper-Log-Log algorithm is used to estimate the number of independent elements in the data stream, and the Count-Min-Sketch algorithm is used to estimate the frequency of each element in the data stream and then the data is deduplicated. After each computing node completes the processing, all data are aggregated using a distributed coordination service (such as Apache-Zoo-Keeper) to obtain global statistical results. The deduplication processing results and the structural score are combined to generate a quality score for the data set. After all data sets are sorted according to the quality score, the sorting results are displayed in real time using the Grafana tool.

[0040] Based on the data sharding strategy, this application distributes the data in the data set to different computing nodes according to key, time, and source, obtains the data volume, data integrity, and matching similarity of the data set, and generates a structural score for the data set. Based on the comparison result of the structural score and the score threshold, it is judged whether it is necessary to perform approximate statistics on the data set. When it is judged that approximate statistics are needed, on each computing node, the Hyper-Log-Log algorithm is used to estimate the number of independent elements in the data stream, and then the Count-Min-Sketch algorithm is used to estimate the frequency of each element in the data stream and then deduplicate the data. This analysis system distributes the collected data to different nodes for parallel processing, and then combines the analysis algorithm to determine whether it is necessary to perform approximate statistics on the data, which ensures the data processing effect while helping to improve data processing efficiency.

[0041] After processing is completed on each computing node, this application aggregates all data using distributed coordination services (such as Apache-Zoo-Keeper) to obtain global statistical results, combines the deduplication processing results with the structural score to generate a quality score for the data set, sorts all data sets according to the quality score, and displays the sorting results in real time through the Grafana tool. This makes it easier for enterprises to manage data sets and improves management efficiency.

[0042] Example 2: Use message queue tools such as Apache-Kafka and RabbitMQ to collect and preprocess data streams to establish a data set. The preprocessing includes data cleaning, format conversion and filtering to ensure data quality and consistency, including the following steps:

[0043] By configuring message queue tools (such as Kafka, RabbitMQ), connecting to different data sources (such as sensors, log files, application data, etc.), ensure that data can flow into the queue system;

[0044] Configure topics in Kafka and queues in RabbitMQ to manage different types of data. Each topic or queue corresponds to a specific type of data stream, and set the number of partitions and replicas of Kafka to ensure high throughput and fault tolerance of the data stream;

[0045] Detect and process missing values ​​in the data. You can choose to fill in default values, interpolate, delete records containing missing values, etc., identify and remove outliers and noise data, which may be caused by sensor failure or human error, detect and remove duplicate records, ensure the uniqueness and accuracy of data, parse raw data into structured formats, such as parsing JSON and XML formatted data into field and value pairs, and convert data to standard units and formats. For example, unify the date format to ISO standards, convert currency units to unified currency units, and ensure that the data types of all fields are consistent, such as converting strings to numeric types for subsequent processing;

[0046] Store the preprocessed data in a persistent storage system, such as a relational database (MySQL, PostgreSQL), a NoSQL database (MongoDB, Cassandra), a distributed file system (HDFS), etc., and create indexes in the persistent storage to speed up queries. For example, create indexes for key fields in a database table.

[0047] Based on the data sharding strategy, the data in the data set is distributed to different computing nodes according to key, time, source, etc., including the following steps:

[0048] The shard key is used to determine the basis for data sharding. You can select a single field or a combination of multiple fields. Common shard keys include:

[0049] Key: Shard based on a unique identifier in the data (such as user ID, order ID).

[0050] Time: Shard based on time fields (such as timestamp, date), suitable for time series data.

[0051] Source: Shard data based on its source (e.g. region, device type, data source system).

[0052] Divide the data by the range of the shard key. For example, data with user IDs in the range of 1 to 1000 is stored in one shard, and data with the range of 1001 to 2000 is stored in another shard. Hash the shard key using a hash function, and then assign the data to different shards. This can evenly distribute the data and avoid overloading certain shards. Assign data to specific shards based on a predefined list. For example, different countries or regions can be assigned to specific shards. Combine multiple sharding strategies and shard based on a combination of multiple fields, such as sharding by time first, and then hashing the key within the time shard.

[0053] Define sharding rules according to the sharding strategy and configure these rules in the system. These rules will guide how to distribute data to different shards, maintain the mapping relationship between sharding keys and computing nodes in the system, ensure that data can be correctly routed to the corresponding nodes, and transmit data streams to various computing nodes in the distributed system through message queue tools (such as Kafka, RabbitMQ). The data received by each node is the shard data corresponding to the node.

[0054] When the amount of data or query load changes, the number of shards can be dynamically increased or decreased. For example, adding new nodes to cope with the growth of data volume, or reducing nodes to save resources when the amount of data decreases. When adjusting the sharding strategy or expanding the shards, data needs to be migrated. This involves moving data from one shard to another while maintaining data consistency and availability.

[0055] After obtaining the data volume, data integrity and matching similarity of the data set, a structural score is generated for the data set. Based on the comparison result between the structural score and the score threshold, it is determined whether approximate statistics of the data set are required, including the following steps:

[0056] The data volume, data integrity and matching similarity are calculated comprehensively to obtain the structure score, which is expressed as: , where Score the structure. is the amount of data, For data integrity, To match similarity, , , are the proportional coefficients of data volume, data integrity and matching similarity, respectively, and , , All are greater than 0;

[0057] The obtained structural score is compared with the preset scoring threshold. The scoring threshold is used to determine whether it is necessary to perform approximate statistics on the data set. If the structural score is less than the scoring threshold, it is determined that it is not necessary to perform approximate statistics on the data set. If the structural score is greater than or equal to the scoring threshold, it is determined that it is necessary to perform approximate statistics on the data set.

[0058] The data volume usually refers to the total number of records in a data set or the total size of the data set. For data stored in a database, you can use SQL query statements to calculate the total number of records. For data stored in a file system, such as CSV or JSON files, you can read the file line by line and count them, or use existing tools (such as the wc -l command) to count the number of rows. You can use the database's metadata query tool to obtain the size of a table or column, and use operating system tools (such as the ls -lh or du -sh command) to view the size of a file or directory. The larger the data volume of a data set, the more approximate statistics are needed to reduce the storage burden of the analysis system to reduce the data volume of the data set.

[0059] The calculation expression of data integrity is: , where For data integrity, is the sum of all missing values ​​in the dataset, is the total number of records in the dataset, is the number of fields in each record. The greater the data completeness, the less missing data in the dataset, that is, the higher the overall quality of the dataset, and the more approximate statistics are needed.

[0060] The calculation logic of matching similarity is: first calculate the Jaccard similarity coefficient between the collected data set and the existing data set in the database, , where is the Jaccard similarity coefficient, Indicates the number of elements in the intersection of the collected dataset and the existing dataset in the database. It represents the number of elements in the union of the collected dataset and the existing dataset in the database, and then selects the maximum Jaccard similarity coefficient as the matching similarity. The expression is: , where To match similarity, Indicates the collection of data sets and database Jaccard similarity coefficient of the data set, Indicates taking the maximum value. The greater the matching similarity, the higher the matching degree between the current collected data set and the existing data set in the enterprise database. That is, the more important the collected data set is to the enterprise, the more approximate statistics are needed.

[0061] When it is determined that approximate statistics are needed, on each computing node, the Hyper-Log-Log algorithm is used to estimate the number of independent elements in the data stream, and the Count-Min-Sketch algorithm is used to estimate the frequency of each element in the data stream and then the data is deduplicated, including the following steps:

[0062] Calculate the hash value for each element in the data stream, update the Hyper-Log-Log register according to the hash value, use the number of leading zeros of the hash value to update the corresponding register, and calculate the estimated value of the number of independent elements. The expression is: , where is an estimated value, is the number of registers, is a constant used to calibrate the registers, It is The value of a register.

[0063] For each element in the data stream, use each hash function to calculate the hash value and increase 1 in the corresponding counter. For the queried element, use all hash functions to calculate the hash value, obtain the estimated frequency from the corresponding counter, and take the minimum value as the final frequency estimate;

[0064] Choose a set of independent hash functions , Represents the number of hash functions. Each hash function maps the data stream element to the index of the counter array. Generally, different hash seeds are selected to generate hash functions. For each element in the data stream, the update steps of Count-Min-Sketch are as follows:

[0065] For each hash function , increase the corresponding counter by 1: , where It is The row counter array is indexed by Counter;

[0066] For the query element, the steps to estimate its frequency are as follows:

[0067] For each hash function , calculate the hash value of the query element and get the index , get the counter value corresponding to each hash function: , the estimated frequency is the minimum of all counter values: , where is the frequency estimate.

[0068] Collect Hyper-Log-Log structures from all computing nodes and merge them to obtain a global estimate of the number of independent elements. Collect the Count-Min-Sketch structures of each computing node and merge them to obtain a global estimate of the element frequency. Based on the frequency estimate of Count-Min-Sketch, determine whether there is duplicate data and delete duplicate data whose frequency estimate is greater than the frequency threshold.

[0069] After each computing node completes processing, all data is aggregated using a distributed coordination service (such as Apache-Zoo-Keeper) to obtain global statistical results, including the following steps:

[0070] Configure the Zoo-Keeper cluster, including the number of nodes, master-slave relationship, synchronization strategy, etc. Design the data node (znode) structure in Zoo-Keeper to store the intermediate and final results of each computing node. When each computing node starts, it registers itself with Zoo-Keeper and reports the status. The registration information includes node ID, status, available resources, etc. Use Zoo-Keeper's monitoring function to track the status of each computing node to ensure the health and availability of nodes during task execution;

[0071] Each computing node stores the local processing results (such as Hyper-Log-Log structure, Count-Min-Sketch structure, etc.) in the designated data node of Zoo-Keeper. The stored information includes the respective statistical results, independent element number estimation, element frequency estimation, etc.

[0072] Ensure that the result data of all computing nodes are synchronized to ZooKeeper. You can use the transactional features of ZooKeeper to ensure data consistency and atomicity. Select one or more dedicated nodes (such as Master nodes) to collect the results of all computing nodes from ZooKeeper and merge them. The merging method depends on the specific algorithm and data structure. Merge the HyperLogLog structure of all nodes to obtain a global estimate of the number of independent elements. Merge the Count-MinSketch structure of all nodes to obtain a global estimate of the element frequency.

[0073] Combine the deduplication processing results with the structure score to generate a quality score for the data set. After sorting all data sets according to the quality score, the sorting results are displayed in real time through the Grafana tool, including the following steps:

[0074] Get the number of deduplication processing times of the dataset on each computing node, sum the number of deduplication processing times on all computing nodes to get the total number of deduplication processing times, normalize the total number of deduplication processing times and the structure score, map the value range of the total number of deduplication processing times and the structure score to [0,1], get the deduplication normalization value and the structure score normalization value, subtract the deduplication normalization value from the structure score normalization value to get the quality score. The larger the quality score, the higher the overall increment of the dataset. Sort all datasets from large to small according to the quality score;

[0075] Configure the database storing the sorting results as a data source for Grafana. Install the corresponding Grafana plugin (such as MySQL, PostgreSQL, Influx-DB, etc.) according to the type of database used. Create a query in Grafana to obtain the sorted data in the database. The query should include the necessary filtering conditions and sorting rules to ensure that the required data is displayed correctly. Create a dashboard in Grafana and add visual components such as tables and charts to display the sorting results. Configure the display mode of the components, including data fields, display styles, refresh frequency, etc. Configure the automatic refresh frequency of the Grafana dashboard to ensure the real-time display of data. Generate a report on the quality score of the data set and the sorting results for internal analysis and decision support.

[0076] Embodiment 3: The big data real-time statistical analysis system based on the distributed computing architecture described in this embodiment includes a data set establishment module, a data distribution module, a data set analysis module, a data deduplication module, and an aggregation display module:

[0077] Dataset creation module: Use message queue tools such as Apache-Kafka and RabbitMQ to collect and preprocess data streams to create data sets. Preprocessing includes data cleaning, format conversion, and filtering to ensure data quality and consistency. The data sets are sent to the data distribution module and the data set analysis module.

[0078] Data distribution module: Based on the data sharding strategy, the data in the data set is distributed to different computing nodes according to key, time, source, etc., and the distribution results are sent to the data deduplication module;

[0079] Dataset analysis module: After obtaining the data volume, data integrity and matching similarity of the data set, a structural score is generated for the data set. Based on the comparison result between the structural score and the scoring threshold, it is determined whether approximate statistics of the data set need to be performed. The judgment result is sent to the data deduplication module, and the structural score is sent to the aggregation display module.

[0080] Data deduplication module: When it is determined that approximate statistics are needed, on each computing node, the Hyper-Log-Log algorithm is used to estimate the number of independent elements in the data stream, and the Count-Min-Sketch algorithm is used to estimate the frequency of each element in the data stream, and then the data is deduplicated. The deduplication results are sent to the aggregation display module;

[0081] Aggregation display module: After each computing node completes processing, all data are aggregated using a distributed coordination service (such as Apache-Zoo-Keeper) to obtain global statistical results. The deduplication processing results and the structural score are combined to generate a quality score for the data set. After all data sets are sorted according to the quality score, the sorting results are displayed in real time through the Grafana tool.

[0082] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0083] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0084] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0085] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0086] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A real-time statistical analysis method for big data based on a distributed computing architecture, characterized by: The analytical method comprises the following steps: Use message queue tools to collect and preprocess data streams to create data sets, and distribute data in the data sets to different computing nodes based on data sharding strategies; After obtaining the data volume, data integrity, and matching similarity of the data set, a structural score is generated for the data set. Based on the comparison result between the structural score and the score threshold, it is determined whether approximate statistics of the data set are required. When it is determined that approximate statistics are required, the number of independent elements in the data stream is estimated by the Hyper-Log-Log algorithm on each computing node, and the frequency of each element in the data stream is estimated by the Count-Min-Sketch algorithm, and then the data is deduplicated. After each computing node completes processing, all data is aggregated using the distributed coordination service to obtain global statistical results. The deduplication processing results and the structural score are combined to generate a quality score for the data set. After all data sets are sorted according to the quality score, the sorting results are displayed in real time through the Grafana tool. After obtaining the data volume, data completeness, and matching similarity of the data set, a structural score is generated for the data set, including the following steps: The data volume, data integrity and matching similarity are calculated comprehensively to obtain the structure score, which is expressed as: , where Score the structure. is the amount of data, For data integrity, To match similarity, , , are the proportional coefficients of data volume, data integrity and matching similarity, respectively, and , , All are greater than 0; Distribute the data in the data set to different computing nodes based on the data sharding strategy, including the following steps: Define sharding rules based on the sharding strategy and configure the rules in the analysis system. The rules are used to guide how to allocate data to different shards. The data stream is transmitted to each computing node in the distributed system through the message queue tool. The data received by each node is the shard data corresponding to the node. When the amount of data or query load changes, dynamically increase or decrease the number of shards, and migrate data when adjusting the sharding strategy or expanding the shards.

2. The big data real-time statistical analysis method based on distributed computing architecture according to claim 1 is characterized by: Based on the comparison results between the structure score and the score threshold, it is determined whether approximate statistics of the data set need to be performed, including the following steps: The obtained structural score is compared with the preset scoring threshold. The scoring threshold is used to determine whether it is necessary to perform approximate statistics on the data set. If the structural score is less than the scoring threshold, it is determined that it is not necessary to perform approximate statistics on the data set. If the structural score is greater than or equal to the scoring threshold, it is determined that it is necessary to perform approximate statistics on the data set.

3. The big data real-time statistical analysis method based on distributed computing architecture according to claim 2 is characterized in that: On each computing node, the Hyper-Log-Log algorithm is used to estimate the number of independent elements in the data stream, and the Count-Min-Sketch algorithm is used to estimate the frequency of each element in the data stream and then perform data deduplication processing, including the following steps: Calculate the hash value for each element in the data stream, update the Hyper-Log-Log register according to the hash value, use the number of leading zeros of the hash value to update the corresponding register, and calculate the estimated value of the number of independent elements. The expression is: , where is an estimated value, is the number of registers, is a constant used to calibrate the registers, It is The value of a register; For each element in the data stream, use each hash function to calculate the hash value and increase 1 in the corresponding counter. For the queried element, use all hash functions to calculate the hash value, obtain the estimated frequency from the corresponding counter, and take the minimum value as the frequency estimate; Collect Hyper-Log-Log structures from all computing nodes and merge them to obtain a global estimate of the number of independent elements. Collect Count-Min-Sketch structures from each computing node and merge them to obtain a global estimate of element frequency. Based on the frequency estimate of Count-Min-Sketch, determine whether there is duplicate data and delete duplicate data whose frequency estimate is greater than the frequency threshold.

4. The big data real-time statistical analysis method based on distributed computing architecture according to claim 3 is characterized by: Combine the deduplication results with the structure score to generate a quality score for the dataset, and sort all datasets according to the quality score, including the following steps: Get the number of deduplication processing times of the dataset on each computing node, sum the number of deduplication processing times on all computing nodes to get the total number of deduplication processing times, normalize the total number of deduplication processing times and the structure score, map the value range of the total number of deduplication processing times and the structure score to [0,1], get the deduplication normalized value and the structure score normalized value, subtract the deduplication normalized value from the structure score normalized value to get the quality score, and sort all datasets from large to small according to the quality score.

5. The big data real-time statistical analysis method based on distributed computing architecture according to claim 4 is characterized in that: The calculation logic of the matching similarity is: first calculate the Jaccard similarity coefficient between the collected data set and the existing data set in the database, , where is the Jaccard similarity coefficient, Indicates the number of elements in the intersection of the collected dataset and the existing dataset in the database. It represents the number of elements in the union of the collected dataset and the existing dataset in the database, and then selects the maximum Jaccard similarity coefficient as the matching similarity. The expression is: , where To match similarity, Indicates the collection of data sets and database Jaccard similarity coefficient of the data set, Indicates taking the maximum value.

6. A big data real-time statistical analysis system based on a distributed computing architecture, used to implement the analysis method according to any one of claims 1 to 5, characterized in that: It includes data set establishment module, data allocation module, data set analysis module, data deduplication module and aggregation display module: Data set building module: Use message queue tools to collect and pre-process data streams to build data sets; Data distribution module: distributes data in the data set to different computing nodes based on the data sharding strategy; Dataset analysis module: After obtaining the data volume, data integrity and matching similarity of the data set, a structural score is generated for the data set. Based on the comparison between the structural score and the score threshold, it is determined whether approximate statistics of the data set are needed; Data deduplication module: When approximate statistics are required, the Hyper-Log-Log algorithm is used on each computing node to estimate the number of independent elements in the data stream, and the Count-Min-Sketch algorithm is used to estimate the frequency of each element in the data stream and then perform data deduplication processing; Aggregation display module: After each computing node completes processing, all data are aggregated using the distributed coordination service to obtain global statistical results. The deduplication processing results and the structural score are combined to generate a quality score for the data set. After all data sets are sorted according to the quality score, the sorting results are displayed in real time through the Grafana tool.