Pre-polymerization storage table creating method

Abstract

The invention discloses a pre-aggregation storage table creation method, which relates to the technical field of data processing, and comprises the following steps: S1, dynamic rule generation: firstly, through a big data analysis system, carrying out deep analysis on a large amount of historical data in an original table, comprehensively mastering data type classification, and analyzing daily average data volume and data time window characteristics; the method comprises the following steps of S1, performing multi-level aggregation window division, S3, performing real-time feedback optimization, S4, performing distributed cooperative processing, S5, performing a self-adaptive compression mechanism, S6, performing a fault-tolerant recovery mechanism, S7, performing data quality monitoring, S8, performing metadata management and S9, performing external system butt joint. Through a dynamic rule generation real-time feedback optimization step and a self-adaptive compression mechanism step, the effects of data processing accuracy, high efficiency and storage cost reduction are achieved, and through a multi-stage aggregation window division distributed cooperative processing step and a fault-tolerant recovery mechanism step, the reliability of data processing is improved. And the effects of enhancing the complex data processing capability of the system, improving the reliability of the system and guaranteeing the service continuity are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for creating a pre-aggregated storage table. Background Art

[0002] With the acceleration of the digitalization process, the scale of data has grown explosively, and the requirements for data processing and storage in various industries have been continuously improved. In the technical field of data processing, pre-aggregated storage tables have become the key to optimizing data management. Taking the Internet of Things and the financial industry as examples, Internet of Things devices generate a large amount of real-time data per second, and financial transaction data requires extremely low processing latency. Traditional data processing methods can no longer meet these needs, and a more efficient method for creating pre-aggregated storage tables is required. This method can preprocess and aggregate the original data, reduce the storage volume, and improve the query and analysis efficiency, playing an important role in enhancing data processing capabilities, reducing storage costs, and supporting real-time decision-making. Therefore, the research and improvement of this method have become an important topic in the field of data processing.

[0003] Existing data processing technologies have deficiencies in many aspects. First, the pre-aggregation rules lack the ability to be dynamically adjusted, making it difficult to adapt to changes in data types, data volumes, and time windows, resulting in an increase in data processing errors and a decrease in efficiency. Second, the data aggregation window division method is single, and it is impossible to accurately process complex data in the time, space, and logical dimensions, affecting the comprehensiveness and accuracy of data aggregation. Third, resource management is poor, lacking real-time monitoring and optimization, and the system is prone to performance degradation due to excessive resource consumption. When performing distributed processing, it is difficult to ensure the consistency of node rules, the central node is under great pressure, and the processing latency is high. In addition, data storage and compression do not fully consider data characteristics, the fault tolerance and recovery mechanism is imperfect, data quality monitoring is not in place, and there are also many inconveniences in metadata management and system docking. These problems limit the efficiency and reliability of data processing, and a method for creating a pre-aggregated storage table needs to be designed to solve the problems raised above. Summary of the Invention

[0004] The purpose of the present invention is to solve the deficiencies existing in the prior art and propose a method for creating a pre-aggregated storage table to solve the problems that limit the efficiency and reliability of data processing in the above technical solutions.

[0005] To achieve the above purpose, the present invention is realized through the following technical solutions: A method for creating a pre-aggregated storage table includes the following creation steps: S1. Dynamic rule generation: First, through a big data analysis system, deeply analyze a large amount of historical data in the original table, comprehensively master the data type classification, and analyze the daily average data volume and the characteristics of the data time window. Then, train the model system through a learning algorithm, optimize the hyperparameters, predict the best pre-aggregation rules, and set the detection threshold for changes in the characteristics of the real-time data stream. When the data model changes by 10% or the data volume fluctuates by more than 20%, re-conduct data analysis, model training, and rule generation; S2. Multi-level aggregation window division: Divide the data generated by the dynamic rules in step S1 through a data processing experience system and a performance test result system into three-level temporary storage windows: First-level time window: Adopt a sliding window mechanism. The initial window size is set to 15 minutes. When the data arrival frequency fluctuates by more than 30%, dynamically adjust the window size according to the frequency change formula; Second-level space window: Divide based on the geographical partition of the data and support cross-regional data aggregation. Ensure accurate data aggregation when the data transmission delay between different regions does not exceed 50 milliseconds; Third-level logic window: Extract data correlation through a semantic analysis tool. For multi-dimensional data including customer transactions, product information, etc., construct at least three layers of multi-dimensional aggregation levels; S3. Real-time feedback optimization: For the three-level temporary storage windows divided by the multi-level aggregation window in step S2, through system monitoring software, combined with Python scripts, monitor the CPU, memory, and I / O resource occupancy rates during the preprocessing process at a frequency of every 10 seconds. When the CPU usage rate exceeds 80%, the memory occupancy rate exceeds 70%, or the I / O read / write delay exceeds 50 milliseconds, based on the monitoring data, use the Q-learning reinforcement learning algorithm. Set the state space as resource metrics such as CPU usage rate, memory occupancy rate, and I / O read / write delay, and the action space as window size adjustment and rule parameter change operations. Through continuous trial-and-error learning, find the optimal strategy to optimize the window division and rule generation strategy; S4. Distributed collaborative processing: Synchronize the pre-aggregation rules among more than 10 nodes using the Paxos consistency protocol for the data monitored and optimized in step S3 to ensure the integrity of data aggregation in a distributed environment. Use blockchain technology to record the rule change history. Each block contains information such as the rule change time and change content to ensure the rule consistency of distributed nodes. At the same time, implement local pre-aggregation through edge computing nodes, and the data volume processed by each edge node accounts for 10% - 20% of the total data volume; S5, Adaptive Compression Mechanism: For the data processed through distributed collaboration in step S4, before selecting a compression algorithm, use the data feature analysis module to perform feature analysis on the pre-aggregation results, including normal distribution and skewed distribution of data, character repetition rate, and data block repetition rate. When the data repetition rate exceeds 30%, select the LZ4 compression algorithm; when the data distribution is relatively uniform, select the Zstandard compression algorithm; S6, Fault Tolerance and Recovery Mechanism: For the data compressed in step S5, through the dual-backup redundancy strategy, the data update of the redundant backup window is kept in real-time synchronization with the main window. Through the heartbeat detection mechanism, it is detected once every 5 seconds. The backup strategy is dynamically adjusted according to the data importance and update frequency. In case of node failure, the uncompleted aggregation tasks can be quickly restored through the redundant backup window, and the recovery time does not exceed 10 minutes; S7, Data Quality Monitoring: For the redundant data in step S6, according to different types of data, formulate a detailed data quality rule library, and perform anomaly detection through real-time comparison. Conduct a quality check on the data every 10 minutes, and mark and record the abnormal data; S8, Metadata Management: For the data detected as abnormal in step S7, use the MySQL relational database to establish and maintain the metadata information of the pre-aggregation storage table, including data structure, aggregation rules, and window division details, and then set indexes to improve query efficiency, ensuring that the metadata query can be completed within 1 second; S9, External System Docking: For the data established and maintained by the relational database in step S8, follow the RESTful API design specification to provide a standardized interface, enabling the pre-aggregation storage table to be seamlessly docked with various external systems, and at the same time, the interface response time does not exceed 10 seconds.

[0006] Furthermore, in the dynamic rule generation step of S1, it specifically includes the following implementation methods: T1, Learning Algorithm Training Model System: By extracting historical data from the data warehouse, after the data undergoes cleaning and normalization preprocessing steps, it is divided into a training set, a validation set, and a test set. Use the TensorFlow framework to build a neural network model, set 3 hidden layers, with the number of nodes in each layer being 256, 128, and 64 respectively, select the ReLU activation function, train for 100 epochs, and predict the best pre-aggregation rules; T2, Trigger the Rule Update Mechanism According to the Feature Changes of the Real-Time Data Stream: Set the feature change monitoring indicators of data type, data volume, and time window. When the number of newly added or decreased data types exceeds 5, the data volume fluctuates by more than 20% continuously for 30 minutes, or the standard deviation of the data distribution within the time window exceeds 0.5, trigger the rule update mechanism, re-collect the real-time data of the past week, and repeat the above learning algorithm training model process.

[0007] Furthermore, in the multi-level window partitioning step of S2, the following implementation methods are specifically included: T3. The time window adopts a sliding window mechanism: The initial time window size is set to 30 minutes, and the sliding step is 5 minutes. By monitoring the data arrival frequency, when the frequency change rate exceeds 40%, the window size is dynamically adjusted according to the formula new-window-size = old-window-size * (1 + frequency-chang-rate); T4. The space window is partitioned based on the geographical partitioning of the data: The globe is divided into 8 geographical regions according to longitude and latitude, and a data aggregation node is set in each region. Through high-speed network connection, it is ensured that the average data transmission delay between different regions does not exceed 30 milliseconds, supporting cross-regional data aggregation; T5. The logical window extracts data relevance through semantic analysis: Using natural language processing technology to perform semantic analysis on text data, for multi-dimensional data including customer behavior, product attributes, etc., a 4-layer multi-dimensional aggregation hierarchy is constructed. The bottom layer is the raw data layer, the second layer is the attribute aggregation layer, the third layer is the relationship aggregation layer, and the top layer is the theme aggregation layer.

[0008] Furthermore, in the distributed collaborative processing step of S4, blockchain technology is used to record the rule change history and local pre-aggregation is achieved through edge computing nodes. When recording the rule change history, the Ethereum blockchain platform is used, and each rule change record is stored as a transaction. The transaction includes the change timestamp, the hash value of the change content, the operation node information, etc. Consensus mechanism is used to ensure data consistency. The blockchain network includes 20 nodes, and the average block generation time is 15 seconds to ensure the rule consistency of distributed nodes; at the same time, in terms of edge computing local pre-aggregation, 50 edge computing nodes distributed in different geographical locations are deployed. Each edge node processes the data of 10 - 20 data sources around it, and a multi-core processor local storage is adopted.

[0009] Furthermore, in the adaptive compression mechanism step of S5, before selecting the compression algorithm, the characteristics such as data distribution and repetition pattern of the pre-aggregation result are analyzed first to improve the compression effect. When analyzing the data distribution, statistical methods are used to calculate statistics such as the data mean, median, and standard deviation to judge the distribution type. For example, when the mean and median of numerical data are close and the standard deviation is small, it is a normal distribution; when the difference is large, it is a skewed distribution. For a normal distribution, the Zstandard algorithm is selected, and for a skewed distribution, the LZ4 or other algorithms are selected according to the skewness direction and degree; when analyzing the repetition pattern, a string matching algorithm is used to detect repeated characters and repeated data blocks. When the character repetition rate exceeds 40% or the data block repetition rate exceeds 30%, the LZ4 algorithm is preferentially selected.

[0010] Further, in the fault tolerance and recovery mechanism steps of S6, the data update of the redundant backup window is kept in real-time synchronization with the main window, and the backup policy is dynamically adjusted according to the data importance and update frequency. The real-time synchronization uses data replication technology to synchronize the main window data to the redundant backup window in real time. A synchronization delay monitoring index is set. When it exceeds 100 milliseconds, the synchronization policy is automatically adjusted to ensure data consistency. In terms of dynamic adjustment of the backup policy, importance levels of 1 to 5 and update frequencies are assigned to the data. Data with importance levels of 4 to 5 and updated more than 10 times per hour are double-backed up in different geographical location backup windows, and data with importance levels of 1 to 3 and low update frequencies are single-backed up in the local backup window.

[0011] Further, in the data quality monitoring steps of S7, corresponding data quality rule libraries are formulated for different types of data and anomaly detection is carried out through real-time comparison. At the same time, in the formulation of the rule library, value range rules are formulated for numerical data, format verification rules are formulated for date-type data, and length limit and character set rules are formulated for string data. The rule library is continuously updated and improved with more than 150 rules. The anomaly detection is carried out every 15 minutes. The data verification tool is used to compare the actual data with the rule library. When it does not conform to the rules, it is marked as abnormal data and the anomaly type, occurrence time and record information of the relevant data are recorded.

[0012] In summary, the present invention provides a method for creating a pre-aggregated storage table, which has the following beneficial effects: 1. Through the dynamic rule generation steps, the real-time feedback optimization steps and the adaptive compression mechanism steps, the accuracy, efficiency of data processing and the reduction of storage costs are achieved. The dynamic rule generation ensures that the pre-aggregation rules fit the data characteristics. The real-time feedback optimization adjusts the strategy dynamically according to the resource occupation to improve the efficiency. The adaptive compression mechanism selects algorithms according to the data characteristics to reduce the storage occupation.

[0013] 2. Through the multi-level aggregation window division steps, the distributed collaborative processing steps and the fault tolerance and recovery mechanism steps, the effects of enhancing the system's ability to process complex data, improving the system's reliability and ensuring business continuity are achieved. The multi-level aggregation windows divide the data according to different dimensions. The distributed collaborative processing realizes multi-node cooperation and ensures the consistency of the rules. The fault tolerance and recovery mechanism quickly recovers tasks in case of node failures.

[0014] 3. Through the steps of training a machine learning model based on historical data, the rule update mechanism step triggered according to the real-time data stream feature changes and the metadata management step, the effects of improving the accuracy of the pre-aggregation rules, enhancing the system's adaptability to data changes and improving the convenience of data management are achieved. The historical data is used to train the model to generate rules, and the rules are updated in real time to adapt to data changes. The metadata management facilitates data query and traceability.

[0015] 4. By adopting the sliding window mechanism step for the time window, the space window based on geographical partitioning step, and the multi-dimensional aggregation hierarchy construction step for the logical window through semantic analysis, the data aggregation quality is comprehensively improved in the time, space, and logical dimensions, providing a high-quality data foundation for in-depth data analysis. Different window mechanisms optimize aggregation from the perspectives of time, space, and data association respectively to meet the complex data analysis requirements. Brief Description of the Drawings

[0016] Figure 1 It is a schematic structural diagram of the process architecture of a method for creating a pre-aggregation storage table according to the present invention. Detailed Embodiments

[0017] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0018] Embodiment: Please refer to Figure 1 As shown, the present invention provides a technical solution: a method for creating a pre-aggregation storage table, including the following creation steps: S1. Dynamic rule generation: First, through a big data analysis system, deeply analyze a large amount of historical data in the original table, comprehensively master the data type classification, analyze the daily average data volume and the characteristics of the data time window, and then train the model system through a learning algorithm. Through hyperparameter tuning, predict the best pre-aggregation rules, and at the same time set the detection threshold for changes in the characteristics of the real-time data stream. When the data model changes by 10% or the data volume fluctuates by more than 20%, re-conduct data analysis, model training, and rule generation. Through accurate analysis of historical data and real-time monitoring, make the pre-aggregation rules highly fit the data characteristics, greatly improve the accuracy and effectiveness of data processing, reduce data processing errors caused by rule mismatches, and improve data processing efficiency; S2. Multi-level aggregation window division: Divide the data generated by the dynamic rule generation in step S1 into three-level temporary storage windows through a data processing experience system and a performance test result system: The first-level time window: Adopt the sliding window mechanism, and the initial window size is set to 15 minutes. When the data arrival frequency fluctuates by more than 30%, dynamically adjust the window size according to the frequency change formula; The second-level space window: Divide based on the geographical partitioning of the data and support cross-regional data aggregation. Ensure accurate data aggregation when the data transmission delay between different regions does not exceed 50 milliseconds; Three - level logic window: Through semantic analysis tools, the data correlation is extracted. For multi - dimensional data including customer transactions, product information, etc., a multi - dimensional aggregation hierarchy with at least three levels is constructed. The multi - level window division can perform targeted processing according to different data characteristics, improve the fineness and comprehensiveness of data aggregation, effectively reduce the complexity of data processing, enhance data processing efficiency, and at the same time ensure the aggregation accuracy of data in different dimensions, providing a high - quality data foundation for subsequent data analysis; S3. Real - time feedback optimization: For the three - level temporary storage window divided by the multi - level aggregation window in step S2, through the system monitoring software and combined with Python scripts, the CPU, memory, and I / O resource occupancy rates during the pre - processing process are monitored and collected at a frequency of every 10 seconds. When the CPU usage rate exceeds 80%, the memory occupancy rate exceeds 70%, or the I / O read - write latency exceeds 50 milliseconds, based on the monitored data, using the Q - learning reinforcement learning algorithm, the state space is set as resource metrics such as CPU usage rate, memory occupancy rate, and I / O read - write latency, and the action space is window size adjustment and rule parameter change operations. Through continuous trial - and - error learning, the optimal strategy is found to optimize the window division and rule generation strategy. Real - time feedback optimization can dynamically adjust the strategy according to the system resource usage situation, avoid system performance degradation caused by excessive resource consumption, save hardware costs, and significantly improve data processing efficiency, ensuring the stable and efficient operation of the system; S4. Distributed collaborative processing: For the data monitored and optimized in step S3, the Paxos consistency protocol is used to synchronize the pre - aggregation rules among more than 10 nodes, ensuring the integrity of data aggregation in a distributed environment. The blockchain technology is used to record the rule change history, and each block contains information such as the rule change time and change content, ensuring the rule consistency of distributed nodes. At the same time, local pre - aggregation is achieved through edge computing nodes, and the data volume processed by each edge node accounts for 10% - 20% of the total data volume. The blockchain technology ensures the traceability and consistency of rule changes, and edge computing reduces the burden on the central node, enabling the system to handle large - scale data processing requirements, reducing data processing latency, and improving the overall performance of the system; S5. Adaptive compression mechanism: For the data processed by distributed collaborative processing in step S4, using the data feature analysis module, before selecting the compression algorithm, the feature analysis of the pre - aggregation result is carried out, including the normal distribution and skewed distribution of data, character repetition rate, and data block repetition rate. When the data repetition rate exceeds 30%, the LZ4 compression algorithm is selected; when the data distribution is relatively uniform, the Zstandard compression algorithm is selected. The adaptive compression mechanism selects the optimal algorithm according to data characteristics, effectively reducing the storage space occupied by data, reducing storage costs, and at the same time ensuring the integrity and efficiency of data during compression and decompression, without affecting the subsequent use of data; S6, Fault Tolerance and Recovery Mechanism: The data compressed in step S5 is redundantly backed up through a dual-backup redundancy strategy. The data update in the redundant backup window is kept in real-time synchronization with the main window. Through a heartbeat detection mechanism, it is detected every 5 seconds. The backup strategy is dynamically adjusted according to the data importance and update frequency. In case of node failure, the uncompleted aggregation tasks are quickly restored through the redundant backup window, and the recovery time does not exceed 10 minutes. The fault tolerance and recovery mechanism greatly improves the data security and business continuity of the system, reduces the risk of data loss and business interruption caused by node failure, ensures the stable operation of the system, and avoids economic losses caused by failures; S7, Data Quality Monitoring: For the redundant data in step S6, a detailed data quality rule library is formulated according to different types of data. Anomaly detection is carried out through real-time comparison. The data quality is checked every 10 minutes. The abnormal data is marked and recorded. The data quality monitoring ensures the accuracy and reliability of the data, provides a reliable basis for data analysis and decision-making, reduces the risk of wrong decisions caused by data quality problems, and enhances the data value and business benefits; S8, Metadata Management: For the data detected as abnormal in step S7, the metadata information of the pre-aggregated storage table, including data structure, aggregation rules, and window division details, is established and maintained using a MySQL relational database. Then, indexes are set to improve the query efficiency, ensuring that the metadata query can be completed within 1 second. The metadata management facilitates users to quickly understand the relevant information of the data, improves the convenience of data management and use, speeds up the data query speed, saves data processing time, and enhances the management efficiency of the data processing process; S9, External System Docking: The data established and maintained in the relational database in step S8 follows the RESTful API design specification to provide a standardized interface, enabling the pre-aggregated storage table to be seamlessly docked with various external systems. At the same time, the interface response time does not exceed 10 seconds. Docking with external systems expands the application scope of the data, realizes the multi-scenario utilization of the data, promotes the maximization of the data value, enhances the business collaboration efficiency, and provides more comprehensive data analysis and decision-making support for the enterprise.

[0019] In the dynamic rule generation step of S1, the following specific implementation methods are included: T1. Learning algorithm training model system: By extracting historical data from the data warehouse, after the data undergoes cleaning and normalization preprocessing steps, it is divided into a training set (accounting for 80%), a validation set (accounting for 10%), and a test set (accounting for 10%). Use the TensorFlow framework to build a neural network model, set 3 hidden layers, with the number of nodes in each layer being 256, 128, and 64 respectively. Select the ReLU activation function, train for 100 epochs, predict the best pre-aggregation rule, train the model based on a large amount of historical data, improve the generalization ability and accuracy of the model, make the generated pre-aggregation rule more in line with the actual data characteristics, thereby improving the quality and efficiency of data processing, and providing a reliable rule basis for subsequent data processing; T2. Trigger the rule update mechanism according to the feature changes of the real-time data stream: Set the feature change monitoring indicators of data type, data volume, and time window. When the number of new or reduced data types exceeds 5, the data volume fluctuates by more than 20% continuously for 30 minutes, or the standard deviation of the data distribution within the time window exceeds 0.5, trigger the rule update mechanism, re-collect the real-time data of the past week, and repeat the above learning algorithm training model process. Monitor and update the rules in real time, so that the pre-aggregation rules can closely follow the dynamic changes of the data, avoid processing errors caused by data changes, improve the real-time performance and accuracy of data processing, and enhance the system's adaptability to complex and changing data environments.

[0020] In the multi-level window division step of S2, it specifically includes the following implementation methods: T3. The time window adopts a sliding window mechanism: The initial time window size is set to 30 minutes, and the sliding step is 5 minutes. By monitoring the data arrival frequency, when the frequency change rate exceeds 40%, the window size is dynamically adjusted according to the formula new-window-size = old-window-size * (1 + frequency-chang-rate). Dynamically adjusting the time window size can better adapt to the changes in the data arrival frequency, improve the rationality and effectiveness of data aggregation in the time dimension, avoid data aggregation deviation caused by unreasonable window size, and improve the data processing accuracy. At the same time, by using the formula to dynamically adjust the window size, the problem of adaptive adjustment of the window size when the data arrival frequency fluctuates is solved, where: new-window-size is the newly dynamically adjusted time window size, old-window-size is the current time window size, and frequency-chang-rate is the change rate of the data arrival frequency; T4. The spatial window is divided based on the geographical partitioning of data: The globe is divided into 8 geographical regions according to longitude and latitude. A data aggregation node is set up in each region and connected through a high-speed network to ensure that the average data transmission delay between different regions does not exceed 30 milliseconds, supporting cross-regional data aggregation. The spatial window is divided based on geographical partitioning, facilitating the targeted processing and aggregation of data at different geographical locations, improving the efficiency and accuracy of cross-regional data processing, and providing strong support for global data integration and analysis. T5. The logical window extracts data relevance through semantic analysis: Using natural language processing technology to perform semantic analysis on text data, for multi-dimensional data including customer behavior, product attributes, etc., a 4-layer multi-dimensional aggregation hierarchy is constructed. The bottom layer is the raw data layer, the second layer is the attribute aggregation layer, the third layer is the relationship aggregation layer, and the top layer is the theme aggregation layer. By constructing a multi-dimensional aggregation hierarchy through semantic analysis, the potential associations between data can be mined, enabling in-depth analysis and aggregation of complex data, providing richer and more valuable information for data analysis, and enhancing the depth and breadth of data analysis.

[0021] In the distributed collaborative processing steps of S4, blockchain technology is used to record the rule change history and local pre-aggregation is achieved through edge computing nodes. When recording the rule change history, the Ethereum blockchain platform is used, and each rule change record is stored as a transaction. The transaction includes the change timestamp, the hash value of the change content, the operation node information, etc. The data consistency is ensured through the consensus mechanism. The blockchain network consists of 20 nodes, and the average block generation time is 15 seconds to ensure the rule consistency of distributed nodes. At the same time, in terms of local pre-aggregation of edge computing, 50 edge computing nodes are deployed at different geographical locations. Each edge node processes the data of 10 - 20 data sources in its vicinity and uses a multi-core processor for local storage. Blockchain technology records the rule change history, ensuring the immutability and traceability of rule changes, enhancing the trust between distributed nodes, ensuring that each node follows consistent rules for data aggregation, and improving the reliability and stability of the system. Edge computing nodes achieve local pre-aggregation, reducing the load on the central node, decreasing the data transmission volume and latency, improving the real-time performance and efficiency of data processing, and enabling the system to better handle large-scale distributed data processing tasks.

[0022] In the steps of the adaptive compression mechanism of S5, before selecting the compression algorithm, the pre-aggregation results are first analyzed for features such as data distribution and repetition patterns to improve the compression effect. When analyzing the data distribution, statistical methods are used to calculate statistics such as the mean, median, and standard deviation of the data to determine the distribution type. For example, if the mean and median of numerical data are close and the standard deviation is small, it is a normal distribution; if the difference is large, it is a skewed distribution. For a normal distribution, the Zstandard algorithm is selected; for a skewed distribution, the LZ4 or other algorithms are selected according to the skewness direction and degree. When analyzing the repetition pattern, the string matching algorithm is used to detect repeated characters and repeated data blocks. When the character repetition rate exceeds 40% or the data block repetition rate exceeds 30%, the LZ4 algorithm is preferentially selected. Selecting the appropriate algorithm according to the data distribution and repetition pattern can give full play to the advantages of different algorithms, improve the compression effect, reduce the data storage space, and at the same time ensure the accuracy and integrity of the data during the compression and decompression processes, without affecting the subsequent analysis and use of the data. Detecting the repetition pattern and selecting the targeted algorithm can effectively improve the compression efficiency of repeated data, further reduce the storage space occupied by the data, reduce the storage cost, and at the same time ensure that the compressed file can be quickly decompressed to meet the requirements of fast data processing.

[0023] In the steps of the fault tolerance and recovery mechanism of S6, the data update of the redundant backup window is kept in real-time synchronization with the main window, and the backup strategy is dynamically adjusted according to the data importance and update frequency. For real-time synchronization, data replication technology is used to synchronize the main window data to the redundant backup window in real-time, and a synchronization delay monitoring index is set. When it exceeds 100 milliseconds, the synchronization strategy is automatically adjusted to ensure data consistency. In terms of the dynamic adjustment of the backup strategy, the data is assigned importance levels and update frequencies from 1 to 5. Data with importance levels 4 to 5 and updated more than 10 times per hour is double-backed up in different geographical location backup windows, and data with importance levels 1 to 3 and low update frequencies is single-backed up in the local backup window. The real-time synchronization mechanism ensures the timeliness and accuracy of the backup data. When a failure occurs in the main window, the backup window can quickly take over the work, reducing the risk of data loss, ensuring the continuity of the business, and avoiding business errors caused by data inconsistency. Dynamically adjusting the backup strategy can reasonably allocate storage resources according to the actual value and update frequency of the data, while ensuring the security of important data, reducing the storage cost, and improving the resource utilization efficiency, achieving the balance between data security and economy.

[0024] In the data quality monitoring steps of S7, corresponding data quality rule libraries are formulated for different types of data, and anomaly detection is carried out through real-time comparison. At the same time, in the formulation of the rule library, a value range rule is formulated for numerical data, a format verification rule is formulated for date-type data, and a length limit and character set rule are formulated for string data. The rule library is continuously updated and improved with more than 150 rules; the anomaly detection is carried out every 15 minutes. The data verification tool is used to compare the actual data with the rule library. When the rules are not met, the data is marked as abnormal data, and the anomaly type, occurrence time, and record information of the relevant data are recorded. The detailed and continuously improved rule library provides a clear standard for data quality monitoring, ensuring that various types of data meet the specification requirements, improving the accuracy and consistency of the data, and providing a reliable data basis for subsequent data analysis and applications; real-time anomaly detection can promptly discover errors and anomalies in the data, facilitating timely processing and correction, preventing incorrect data from entering the subsequent processing process, and reducing business risks and losses caused by data quality problems.

[0025] The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention in any other form. Any person skilled in the art may use the technical content disclosed above to make changes or modifications into equivalent embodiments with equivalent changes and apply them to other fields. However, as long as the technical solution content of the present invention is not departed from, any simple modification, equivalent change, and modification made to the above embodiments based on the technical essence of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A method for creating a pre-aggregation storage table, characterized in that: It includes the following creation steps: S1. Dynamic rule generation: First, through a big data analysis system, deeply analyze a large amount of historical data in the original table, comprehensively master the data type classification, analyze the daily average data volume and the characteristics of the data time window. Then, train the model system through a learning algorithm, optimize the hyperparameters, predict the best pre-aggregation rules, and set the detection threshold for changes in the characteristics of the real-time data stream. When the data model changes by 10% or the data volume fluctuates by more than 20%, re-conduct data analysis, model training, and rule generation; S2. Multi-level aggregation window division: Divide the data generated by the dynamic rule generation in step S1 through a data processing experience system and a performance test result system into three levels of temporary storage windows: First-level time window: Adopt a sliding window mechanism, set the initial window size to 15 minutes, and dynamically adjust the window size according to the frequency change formula when the data arrival frequency fluctuates by more than 30%; Second-level space window: Divide it based on the geographical partition of the data and support cross-regional data aggregation. Ensure accurate data aggregation when the data transmission delay between different regions does not exceed 50 milliseconds; Third-level logical window: Extract data relevance through a semantic analysis tool, and construct at least three levels of multi-dimensional aggregation hierarchies for multi-dimensional data including customer transactions, product information, etc.; S3. Real-time feedback optimization: For the three levels of temporary storage windows divided by the multi-level aggregation window in step S2, use system monitoring software to combine Python scripts to monitor the CPU, memory, and I / O resource occupancy rates during the preprocessing process at a frequency of every 10 seconds. When the CPU usage rate exceeds 80%, the memory occupancy rate exceeds 70%, or the I / O read / write delay exceeds 50 milliseconds, based on the monitoring data, use the Q-learning reinforcement learning algorithm, set the state space as resource metrics such as CPU usage rate, memory occupancy rate, and I / O read / write delay, and the action space as window size adjustment and rule parameter change operations. Through continuous trial-and-error learning, find the optimal strategy to optimize the window division and rule generation strategy; S4. Distributed collaborative processing: Synchronize the pre-aggregation rules among more than 10 nodes using the Paxos consistency protocol for the data monitored and optimized in step S3 to ensure the integrity of data aggregation in a distributed environment. Use blockchain technology to record the rule change history, and each block contains information such as the rule change time and change content to ensure the rule consistency of distributed nodes. At the same time, achieve local pre-aggregation through edge computing nodes, and the data volume processed by each edge node accounts for 10% - 20% of the total data volume; S5. Adaptive compression mechanism: For the data processed by distributed collaborative processing in step S4, use a data feature analysis module to perform feature analysis on the pre-aggregation results before selecting a compression algorithm, including data normal distribution and skewed distribution, character repetition rate, and data block repetition rate. When the data repetition rate exceeds 30%, select the LZ4 compression algorithm; when the data distribution is relatively uniform, select the Zstandard compression algorithm; S6, Fault Tolerance and Recovery Mechanism: The data compressed in step S5 is redundantly backed up through a dual-backup redundancy strategy. The data update in the redundant backup window is kept in real-time synchronization with the main window. Through a heartbeat detection mechanism, it is detected once every 5 seconds. The backup strategy is dynamically adjusted according to the data importance and update frequency. In case of node failure, the uncompleted aggregation tasks are quickly restored through the redundant backup window, and the recovery time does not exceed 10 minutes. S7, Data Quality Monitoring: For the redundant data in step S6, a detailed data quality rule library is formulated according to different types of data. Anomaly detection is carried out through real-time comparison. The data quality is checked every 10 minutes, and the abnormal data is marked and recorded. S8, Metadata Management: For the data detected as abnormal in step S7, the metadata information of the pre-aggregation storage table is established and maintained using a MySQL relational database, including data structure, aggregation rules, and window division details. Then, indexes are set to improve the query efficiency, ensuring that the metadata query can be completed within 1 second. S9, External System Docking: The data established and maintained in the relational database in step S8 follows the RESTful API design specification to provide a standardized interface, enabling the pre-aggregation storage table to be seamlessly docked with various external systems. At the same time, the interface response time does not exceed 10 seconds.

2. The method for creating a pre-aggregation storage table according to claim 1, wherein: In the dynamic rule generation step of S1, the following specific implementation methods are included: T1, Learning Algorithm Training Model System: By extracting historical data from the data warehouse, after the data undergoes cleaning and normalization preprocessing steps, it is divided into a training set, a validation set, and a test set. A neural network model is built using the TensorFlow framework, with 3 hidden layers, and the number of nodes in each layer is 256, 128, and 64 respectively. The ReLU activation function is selected, and it is trained for 100 epochs to predict the optimal pre-aggregation rules. T2, Trigger the Rule Update Mechanism According to the Feature Changes of the Real-Time Data Stream: Set the monitoring indicators for feature changes of data type, data volume, and time window. When the number of newly added or reduced data types exceeds 5, the data volume fluctuates by more than 20% continuously for 30 minutes, or the standard deviation of the data distribution within the time window exceeds 0.5, trigger the rule update mechanism, collect the real-time data of the past week again, and repeat the above learning algorithm training model process.

3. A method for creating a pre-aggregation storage table according to claim 1, characterized in that: In the multi-level window division step of S2, the following specific implementation methods are included: T3, The time window adopts a sliding window mechanism: The initial time window size is set to 30 minutes, and the sliding step is 5 minutes. By monitoring the data arrival frequency, when the frequency change rate exceeds 40%, the window size is dynamically adjusted according to the formula new-window-size = old-window-size * (1 + frequency-chang-rate). T4, The spatial window is divided based on the geographical partition of the data: The globe is divided into 8 geographical regions according to longitude and latitude, and a data aggregation node is set in each region. Through a high-speed network connection, it is ensured that the average data transmission delay between different regions does not exceed 30 milliseconds, supporting cross-regional data aggregation. T5. The logical window extracts data relevance through semantic analysis: Using natural language processing technology to perform semantic analysis on text data, for multi-dimensional data including customer behavior, product attributes, etc., a 4-layer multi-dimensional aggregation hierarchy is constructed. The bottom layer is the original data layer, the second layer is the attribute aggregation layer, the third layer is the relationship aggregation layer, and the top layer is the theme aggregation layer.

4. A method for creating a pre-aggregation storage table according to claim 1, characterized in that: In the distributed collaborative processing step of S4, blockchain technology is used to record the rule change history and local pre-aggregation is achieved through edge computing nodes. When recording the rule change history, the Ethereum blockchain platform is used to store each rule change record as a transaction. The transaction includes the change timestamp, the hash value of the change content, the operation node information, etc. The consensus mechanism is used to ensure data consistency. The blockchain network contains 20 nodes, and the average block generation time is 15 seconds to ensure the rule consistency of distributed nodes. At the same time, in terms of local pre-aggregation of edge computing, 50 edge computing nodes distributed in different geographical locations are deployed. Each edge node processes the data of 10 - 20 data sources in its vicinity and uses a multi-core processor for local storage.

5. A method for creating a pre-aggregated storage table according to claim 1, characterized in that: In the adaptive compression mechanism step of S5, before selecting the compression algorithm, the pre-aggregation result is first analyzed for features such as data distribution and repetition patterns to improve the compression effect. When analyzing the data distribution, statistical methods are used to calculate statistics such as the data mean, median, and standard deviation to judge the distribution type. When analyzing the repetition pattern, a string matching algorithm is used to detect repeated characters and repeated data blocks. When the character repetition rate exceeds 40% or the data block repetition rate exceeds 30%, the LZ4 algorithm is preferentially selected.

6. A method for creating a pre-aggregation storage table according to claim 1, characterized in that: In the fault tolerance and recovery mechanism step of S6, the data update of the redundant backup window is kept in real-time synchronization with the main window, and the backup strategy is dynamically adjusted according to the data importance and update frequency. Real-time synchronization uses data replication technology to synchronize the main window data to the redundant backup window in real-time. A synchronization delay monitoring index is set. When it exceeds 100 milliseconds, the synchronization strategy is automatically adjusted to ensure data consistency. Regarding the dynamic adjustment of the backup strategy, importance levels from 1 to 5 and update frequencies are assigned to the data. Data with importance levels 4 - 5 and updated more than 10 times per hour is double-backed up in different geographical location backup windows, and data with importance levels 1 - 3 and low update frequencies is single-backed up in the local backup window.

7. A method for creating a pre-aggregation storage table according to claim 1, characterized in that: In the data quality monitoring step of S7, corresponding data quality rule libraries are formulated for different types of data and anomaly detection is performed through real-time comparison. At the same time, in formulating the rule library, value range rules are formulated for numerical data, format verification rules are formulated for date data, and length limit and character set rules are formulated for string data. The rule library is continuously updated and improved with more than 150 rules. Anomaly detection is performed every 15 minutes. A data verification tool is used to compare the actual data with the rule library. When the rules are not met, the data is marked as abnormal data and the anomaly type, occurrence time, and record information of the relevant data are recorded.

Images