System and operation method for centralized acquisition and filling of data cells in dimension tree
By using data models and architecture, parallel query and calculation modules, data verification and fill modules in the data cell central acquisition and filling system of the dimension tree, performance bottlenecks and consistency problems in large-scale data processing are solved, and efficient , reliable data processing and filling effects.
Patent Information
- Application Number
- CN202411178342.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-27
AI Technical Summary
When working with large-scale data sets, centralized acquisition and filling of data cells of dimension trees can lead to performance bottlenecks, data consistency issues, and increased difficulty and error risks of complex structures.
A centralized acquisition and filling system for the dimension tree is designed, including data model and architecture, parallel query and calculation module, data verification module, data filling module and system monitoring and maintenance module, using cache strategy, data sharding and partitioning scheme, Multithreaded or multi-process technology, weighted average computing, locking mechanism, transaction processing and asynchronous strategies ensure data accuracy and consistency, and adjust system performance through real-time monitoring.
Significantly improves the efficiency and scalability of data access and updates, reduces computing resource consumption, ensures data accuracy and consistency, improves system availability, response speed and throughput, and enhances system stability and reliability. sex.
Smart Images

Figure CN118689906B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of cell data processing technology, and specifically to a system and operation method for centralized acquisition and filling of data cells of a dimension tree. Background Art
[0002] Data cell filling usually refers to entering or updating data in a specific location (i.e., cell) in a table (such as Excel, Google Sheets, etc.) or database (such as MySQL, MongoDB, etc.). This data can be text, numbers, dates, times, etc., depending on the purpose of the cell and the requirements of the data structure.
[0003] For example, publication number CN110598194B discloses a method, device and terminal device for extracting content from a non-full grid table. The method includes: obtaining an original table, which is a non-full grid table; reading the original table to obtain a full grid table, and determining each cell belonging to the same merged cell; obtaining the data of each cell from the full grid table in turn, determining whether the cell with non-empty data belongs to a title cell or a content cell, and marking the title cell and the content cell; if the merged cell includes a title cell, then each cell belonging to the merged cell is filled with the data of the title cell. Through this embodiment of the invention, the non-full grid table is regularized into a full grid table, so that the data of each cell can be accurately read out, and the title cell and the content cell are marked.
[0004] As shown in the above technology, the existing technology is still simply extracting and filling the cell data. In actual use, especially in dealing with complex data cells based on dimension trees, there are still the following problems:
[0005] Performance bottleneck: When processing large-scale data sets, centralized acquisition and filling of data may consume a lot of computing resources and time. Improper allocation of resources will affect system performance.
[0006] Data consistency: In a distributed system, multiple nodes may try to update the same data cell at the same time, which may lead to data inconsistency. In addition, if data is erroneous or lost during transmission, it may also affect the final consistency of the data;
[0007] Complexity: The structure of the dimension tree can be very complex, containing multiple levels and different types of nodes. This increases the difficulty and complexity of centrally acquiring and filling data cells, which makes it easy to make mistakes. Summary of the invention
[0008] In view of the deficiencies of the prior art, the present invention provides a system and an operation method for centralized acquisition and filling of data cells of a dimension tree, which solves the problems of the prior art.
[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions: a system for centralized acquisition and filling of data cells of a dimension tree, comprising:
[0010] Data model and architecture, used to design the data model of the dimension tree, clarify the storage structure and access method of each cell; and plan the system architecture, design cache strategies to speed up the reading of frequently accessed cells, and determine the data sharding and partitioning scheme to allocate the access and update tasks of the dimension tree cells;
[0011] Data preprocessing module, used to clean the imported data source data and remove outliers;
[0012] The parallel query and calculation module uses multi-threading or multi-process technology to query and obtain the data of dimension tree cells in parallel, and combines incremental aggregation and data compression rate algorithms to reduce the amount of calculation;
[0013] Data validation module, which verifies the accuracy and legitimacy of data using checksums or hash values before populating the data into dimension tree cells;
[0014] The data filling module fills the verified data into the corresponding cells of the dimension tree. The weighted average is used for calculation during the filling process, and the filling is performed according to the rules of the set update strategy. During the filling process, a lock mechanism or transaction processing is used to ensure the data consistency when multiple threads or processes update the same cell. For the update of non-critical cells, an asynchronous strategy is used to improve the system's response speed and throughput;
[0015] The system monitoring and maintenance module monitors the system's performance indicators and flexibly adjusts the acquisition, processing and filling parameters of the dimension tree cell data to maintain the system performance indicators within the safety threshold.
[0016] The present invention also discloses an operation method of a system for centrally acquiring and filling data cells of a dimension tree. Before data acquisition and filling, the following setting steps are first performed:
[0017] A1. Clarify the data source DS of the project and its characteristics, including data format, size and update frequency;
[0018] A2. Determine the specific position of the target fill position TP in the dimension tree and the expected format FG;
[0019] A3. Analyze the structure of the dimension tree and identify the number of nodes N and the number of levels L.
[0020] Preferably, the operation method of the data model and architecture specifically includes:
[0021] B1. When designing the data model of the dimension tree, make sure each cell i,j The storage structure and access method of the node are as follows: i represents the row / level and j represents the column / attribute; and the node storage information is designed, including the node ID and the attribute set A;
[0022] B2. When planning the system architecture, design a cache strategy to speed up the reading of frequently accessed cells, considering the cache size C SIZE With the expected data access pattern, the cache size C SIZE It will affect the cache hit rate HR. When the cache hit rate HR is improved, the number of accesses to the original data source will be reduced, thereby reducing the processing time T. P , the cache hit rate HR calculation formula is: ;
[0023] Among them, T0 represents the number of hits, T 总 Indicates the total number of requests;
[0024] B3. At the same time, determine the number of data shards DP and the partitioning strategy PART to evenly distribute the access and update tasks of the dimension tree cells during parallel processing.
[0025] Preferably, the operation method of the data preprocessing module specifically includes:
[0026] C1. Import the data source DS, extract the data related to the dimension tree cells, and obtain the clean data set DCS after cleaning. During the cleaning process, the threshold TO is used to determine and remove outliers.
[0027] Preferably, the operation method of the parallel query and calculation module specifically includes:
[0028] D1. Use multi-threading or multi-process technology to query and obtain the data of dimension tree cells from the data source or cache in parallel, and combine incremental aggregation IA technology for calculation optimization and data compression technology for compression processing;
[0029] D2. In terms of computing optimization, incremental aggregation IA is used to reduce repeated computing. The calculation of incremental aggregation depends on the result of the previous state, that is:
[0030] Σ new = Σ old + Δ;
[0031] where Σ old represents the aggregate value before adding new data or changing; Δ represents the increment or change, that is, the change in the aggregate value caused by the addition of new data or the change of existing data; Σ newRepresents the updated aggregate value by converting the old aggregate value Σ old Add to the increment Δ to get;
[0032] The data compression ratio CR calculation formula is: Among them, C 前 The original data size before compression, C 后 Indicates the size of the compressed data.
[0033] Preferably, the operation method of the data verification module specifically includes:
[0034] E1. Source data verification: Before submitting data to the dimension tree, verify the checksum / hash value of the source data to ensure that the data has not been tampered with during transmission or preprocessing;
[0035] E2. Verification of existing data in the dimension tree: If the dimension tree already contains some data, compare the checksum / hash value of the new data with the checksum / hash value of the stored data to check for potential duplication or conflict;
[0036] E3. Cross-cell consistency check: For complex data relationships involving multiple cells, additional logic checks are implemented to ensure data consistency, including verifying whether the associations between different dimensions are correct and whether the calculated summary values match the underlying data;
[0037] E4. Recording and reporting: When data inconsistencies are found, relevant details are recorded immediately, including inconsistent data points, timestamps, cells involved, and detailed reports are generated;
[0038] E5. Problem tracing: trace back the source and processing of data to find the root cause of inconsistency;
[0039] E6. Data correction: Depending on the nature of the problem, correct the erroneous data or update the data processing logic to ensure future data consistency;
[0040] E7. Revalidation: Revalidate the corrected data to ensure that all issues have been resolved and the consistency and accuracy of the data have been restored.
[0041] Preferably, the operation method of the data filling module specifically includes:
[0042] F1. Fill the verified data into the corresponding cells of the dimension tree. The weighted average calculation method is used in the filling process. The calculation formula of the weighted average is: ;W m is the weight, X m is the value, m represents the mth data point;
[0043] F2. Implement update strategies, including direct coverage, incremental updates, or merged updates based on different business needs;
[0044] F3. During the filling process, use locking mechanisms or transaction processing to ensure data consistency when multiple threads or processes update the same cell;
[0045] F4. For updates of non-critical cells, an asynchronous strategy is used to improve the system's response speed and throughput;
[0046] F5. When performing performance testing, evaluate the processing time T PROCESS , throughput THROUGHPUT indicator.
[0047] Preferably, the data filling step includes:
[0048] F3.1. Locate the target position TP: Use the structural information of the dimension tree and the path information of the target filling position TP to locate the specific filling position in the dimension tree;
[0049] F3.2, Data formatting and filling: Format the pre-processed data according to the target format FG, and fill the formatted data into the target position TP;
[0050] F3.3, Verification and Feedback:
[0051] F3.3.1. Verify that the data is successfully filled into the target location and check the integrity and accuracy of the data;
[0052] F3.3.2. Feedback the filling results to the system administrator or user as needed, including success information, error logs or alarm notifications.
[0053] Preferably, the operation method of the system monitoring and maintenance module specifically includes:
[0054] G1. Monitor system performance indicators, including CPU usage and memory usage;
[0055] G2. Adjust cache size according to monitoring data C SIZE 、Number of threads T THREADS , parameter of the number of shards DP;
[0056] G3. Write automated test scripts to cover different data processing scenarios and perform system performance monitoring and parameter adjustment tests.
[0057] Preferably, the cache size C is adjusted SIZE 、Number of threads T THREADS The steps to adjust the parameters of the number of shards DP are:
[0058] G2.1. Set the CPU usage threshold to YCPU , the memory usage threshold is Y MTU , and record CPU usage, memory usage, throughput, and cache hit rate indicators;
[0059] G2.2, Cache size C SIZE Adjustment:
[0060] If the cache hit rate C SIZE Low: Increase cache size C SIZE , to reduce access to backend storage;
[0061] If memory usage approaches or exceeds the threshold: Reduce cache size C SIZE ;
[0062] G2.3, number of threads T THREADS Adjustment:
[0063] If CPU usage is less than half of the threshold: increase the number of threads T THREADS , increase throughput;
[0064] If CPU usage approaches or exceeds the threshold: reduce the number of threads T THREADS ;
[0065] G2.4, Shard Quantity DP Adjustment:
[0066] If the system responds slowly when processing large data sets: increase the number of shards DP to distribute the processing pressure;
[0067] If the CPU and memory utilization of system resources are unbalanced: gradually adjust the number of shards DP until the resources are used in a balanced manner;
[0068] G2.5, Iterative adjustment and testing:
[0069] Repeat the test: After each parameter adjustment, rerun the performance test and record the new performance indicator data;
[0070] Evaluation effect: Compare the performance indicators before and after the adjustment to evaluate whether the adjustment is effective;
[0071] Continuous monitoring: After parameter adjustment, the system status is continuously monitored to ensure that the system operates stably within the threshold range.
[0072] The present invention provides a system and method for centrally acquiring and filling data cells of a dimension tree. Compared with the prior art, it has the following beneficial effects:
[0073] 1. The centralized data cell acquisition and filling system of the dimension tree significantly improves the efficiency and scalability of data access and update through carefully designed data models and architectures, especially cache strategies, data sharding and partitioning schemes. Secondly, the introduction of parallel query and calculation modules, combined with incremental aggregation and data compression technology, greatly reduces the consumption of computing resources and improves the processing speed. Furthermore, the addition of the data verification module ensures the accuracy and legality of the data and enhances the reliability of the system. In addition, the weighted average calculation and flexible update strategy in the data filling module, combined with the lock mechanism or transaction processing, effectively ensure data consistency, while the asynchronous strategy improves the efficiency of non-critical cell updates. Finally, the real-time adjustment capability of the system monitoring and maintenance module ensures the stability and optimization of system performance, and improves the overall system availability, response speed and throughput.
[0074] 2. The operation method of the centralized acquisition and filling system of the data cells of the dimension tree enhances the data adaptability and flexibility of the system by analyzing the data source characteristics and target filling positions in detail, so that the system can handle different types of data sources and target formats more efficiently. Secondly, the refinement of the data model design, especially the clarification of the cell storage structure and access method, as well as the detailed planning of node storage information, provides a solid foundation for subsequent data processing and improves the accuracy and efficiency of data processing. At the system architecture level, the introduction of caching strategies, data sharding and partitioning strategies significantly improves the system's ability to handle high-frequency access cells and reduces dependence on the original data source. At the same time, balanced distribution of tasks is achieved through parallel processing, which improves the overall performance and scalability of the system.
[0075] 3. The operation method of the centralized acquisition and filling system of the data cells of the dimension tree. The data preprocessing effectively improves the data quality through a sophisticated cleaning process combined with thresholds to determine abnormal values, laying a solid foundation for subsequent processing. The parallel query and calculation module significantly reduces the computational complexity and storage requirements through multi-threading / multi-process technology combined with incremental aggregation and data compression technology. Incremental aggregation reduces repeated calculations and improves processing efficiency; data compression further reduces the space occupied by data and speeds up data transmission and processing. These improvements jointly improve the overall performance and data processing efficiency of the system, providing users with faster and more accurate data services.
[0076] 4. The operation method of the centralized acquisition and filling system of the data cells of the dimension tree significantly improves the accuracy and reliability of the data through multiple verification methods such as source data verification, existing data verification in the dimension tree, and cross-cell consistency check. The recording and reporting mechanism and problem tracing capabilities enable data problems to be discovered and effectively resolved in a timely manner. The closed-loop process of data correction and re-verification ensures the continuous restoration of data consistency and accuracy, providing users with higher quality data assurance. These improvements together improve the credibility of data processing and the overall performance of the system.
[0077] 5. The operation method of the centralized acquisition and filling system of data cells in the dimension tree introduces a weighted average calculation method and a flexible update strategy, which improves the accuracy and flexibility of data filling. At the same time, data consistency is ensured through a lock mechanism or transaction processing, avoiding conflicts during concurrent updates. The asynchronous update strategy optimizes the update efficiency of non-critical cells and improves the system response speed and throughput. The introduction of performance testing provides a basis for system performance tuning, further improving the overall data processing capabilities and user experience. These improvements together enhance the robustness and efficiency of the data filling module.
[0078] 6. The operation method of the centralized acquisition and filling system of the data cells of the dimension tree significantly improves the efficiency and accuracy of system performance tuning by real-time monitoring of performance indicators and automatically adjusting the cache, number of threads and number of shards based on preset thresholds. This automated adjustment mechanism can not only quickly respond to performance bottlenecks, but also find the best balance between resource utilization and performance, reduce manual intervention, and improve system stability and maintainability. At the same time, the introduction of automated test scripts ensures the effectiveness of adjustments, forms a closed-loop optimization process, and continuously ensures the efficient operation of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0079] Figure 1 It is a system principle block diagram of the present invention;
[0080] Figure 2 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION
[0081] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0082] See also Figure 1-Figure 2 , the present invention provides the following two technical solutions:
[0083] Embodiment 1: A system for centralized acquisition and filling of data cells of a dimension tree, comprising:
[0084] Data model and architecture, used to design the data model of the dimension tree, clarify the storage structure and access method of each cell; and plan the system architecture, design cache strategies to speed up the reading of frequently accessed cells, and determine the data sharding and partitioning scheme to allocate the access and update tasks of the dimension tree cells;
[0085] Data preprocessing module, used to clean the imported data source data and remove outliers;
[0086] The parallel query and calculation module uses multi-threading or multi-process technology to query and obtain the data of dimension tree cells in parallel, and combines incremental aggregation and data compression rate algorithms to reduce the amount of calculation;
[0087] Data validation module, which verifies the accuracy and legitimacy of data using checksums or hash values before populating the data into dimension tree cells;
[0088] Checksum is a simple method for detecting the integrity of data after it has been transmitted or stored. It does this by calculating some form of summary or fingerprint of the data and transmitting or storing it together with the original data. The receiver or retriever can recalculate the checksum of the data using the same algorithm and compare it with the received checksum to verify that the data has not been altered during transmission or storage.
[0089] A hash value, also known as a hash value, hash value or message digest, is a function that maps input data of arbitrary length (called a message or plaintext) to output data of a fixed length (called a hash value or ciphertext).
[0090] The data filling module fills the verified data into the corresponding cells of the dimension tree. The weighted average is used for calculation during the filling process, and the filling is performed according to the rules of the set update strategy. During the filling process, a lock mechanism or transaction processing is used to ensure the data consistency when multiple threads or processes update the same cell. For the update of non-critical cells, an asynchronous strategy is used to improve the system's response speed and throughput;
[0091] The system monitoring and maintenance module monitors the system's performance indicators and flexibly adjusts the acquisition, processing and filling parameters of the dimension tree cell data to maintain the system performance indicators within the safety threshold.
[0092] Through the carefully designed data model and architecture, especially the cache strategy, data sharding and partitioning scheme, the efficiency and scalability of data access and update are significantly improved. Secondly, the introduction of parallel query and calculation modules, combined with incremental aggregation and data compression technology, greatly reduces the consumption of computing resources and improves the processing speed. Furthermore, the addition of the data verification module ensures the accuracy and legality of the data and enhances the reliability of the system. In addition, the weighted average calculation and flexible update strategy in the data filling module, combined with the lock mechanism or transaction processing, effectively ensure data consistency, while the asynchronous strategy improves the efficiency of non-critical cell updates. Finally, the real-time adjustment capability of the system monitoring and maintenance module ensures the stability and optimization of system performance, and improves the overall system availability, response speed and throughput.
[0093] Embodiment 2: The present invention also discloses an operation method of a system for centralized acquisition and filling of data cells of a dimension tree. Before data acquisition and filling, the following setting steps are first performed:
[0094] A1. Clarify the data source DS of the project and its characteristics, including data format, size and update frequency;
[0095] A2. Determine the specific location of the target fill position TP in the dimension tree and the expected format FG, such as JSON, XML or a custom format;
[0096] A3. Analyze the structure of the dimension tree, identify the number of nodes N and the number of levels L, and provide a basis for data model design.
[0097] The specific methods of operating the data model and architecture include:
[0098] B1. When designing the data model of the dimension tree, make sure each cell i,j The storage structure and access method of the data structure are as follows: i represents rows / levels and j represents columns / attributes; suppose there is a dimension tree of sales data, which is layered by year, quarter, product and region, where level i represents year, quarter, product and region, and attribute j represents sales amount, sales volume, etc. The node storage information is designed, including node ID and attribute set A. The accurate storage of this information is the basis for subsequent query and calculation; suppose there is a node representing "the sales of product A in North China in Q1 2023", and its node ID can be an automatically generated UUID or a string like "2023_Q1_Product A_North China", and the attribute set A of the node may contain the following entries:
[0099] "Sales": 10000;
[0100] "sales": 500;
[0101] B2. When planning the system architecture, design a cache strategy to speed up the reading of frequently accessed cells, considering the cache size C SIZE With the expected data access pattern, such as frequently accessed data should be cached first, the cache size C SIZE It will affect the cache hit rate HR. When the cache hit rate HR is improved, the number of accesses to the original data source will be reduced, thereby reducing the processing time T. P , the cache hit rate HR calculation formula is:
[0102] ;
[0103] Among them, T0 represents the number of hits, T 总 Indicates the total number of requests.
[0104] B3. At the same time, determine the number of data shards DP and the partition strategy PART to evenly distribute the access and update tasks of the dimension tree cells during parallel processing; the selection of the number of shards DP and the partition strategy PART will affect the parallelism and load balancing of data queries, and thus affect the throughput THROUGHPUT.
[0105] By analyzing the characteristics of the data source and the target filling position in detail, the data adaptability and flexibility of the system are enhanced, enabling the system to handle different types of data sources and target formats more efficiently. Secondly, the refinement of the data model design, especially the clarification of the cell storage structure and access method, as well as the detailed planning of the node storage information, provides a solid foundation for subsequent data processing and improves the accuracy and efficiency of data processing. At the system architecture level, the introduction of caching strategies, data sharding and partitioning strategies significantly improves the system's ability to handle high-frequency access cells and reduces dependence on the original data source. At the same time, balanced distribution of tasks is achieved through parallel processing, improving the overall performance and scalability of the system.
[0106] The operation method of the data preprocessing module specifically includes:
[0107] C1. Import the data source DS, extract the data related to the dimension tree cells, and obtain the clean data set DCS after cleaning. During the cleaning process, the threshold TO is used to determine and remove outliers.
[0108] The operation method of the parallel query and calculation module specifically includes:
[0109] D1. Use multi-threading or multi-process technology to query and obtain the data of dimension tree cells from the data source or cache in parallel, and combine incremental aggregation IA technology for calculation optimization and data compression technology for compression processing;
[0110] Incremental aggregation is a data processing technology that is particularly useful when processing large-scale data streams. This technology avoids repeated calculations on the entire data set by only calculating and aggregating the newly added data, thereby improving computing efficiency and response speed. Incremental aggregation is widely used in many fields such as real-time data analysis, real-time monitoring, and real-time recommendation.
[0111] D2. In terms of computing optimization, incremental aggregation IA is used to reduce repeated computing. The calculation of incremental aggregation depends on the result of the previous state, that is: Σ new = Σ old + Δ; where Σ old represents the aggregate value before adding new data or changing; Δ represents the increment or change, that is, the change in the aggregate value caused by the addition of new data or the change of existing data; Σ new Represents the updated aggregate value by converting the old aggregate value Σ old Add to the increment Δ to get;
[0112] The data compression ratio CR calculation formula is: Among them, C 前 The original data size before compression, C 后 Indicates the size of the compressed data.
[0113] Data preprocessing effectively improves data quality through a sophisticated cleaning process combined with threshold determination of outliers, laying a solid foundation for subsequent processing. The parallel query and calculation module significantly reduces computational complexity and storage requirements through multi-threading / multi-process technology combined with incremental aggregation and data compression technology. Incremental aggregation reduces repeated calculations and improves processing efficiency; data compression further reduces the space occupied by data and speeds up data transmission and processing. These improvements together improve the overall performance and data processing efficiency of the system, providing users with faster and more accurate data services.
[0114] The operation method of the data verification module specifically includes:
[0115] E1. Source data verification: Before submitting data to the dimension tree, verify the checksum / hash value of the source data to ensure that the data has not been tampered with during transmission or preprocessing;
[0116] E2. Verification of existing data in the dimension tree: If the dimension tree already contains some data, compare the checksum / hash value of the new data with the checksum / hash value of the stored data to check for potential duplication or conflict;
[0117] E3. Cross-cell consistency check: For complex data relationships involving multiple cells, additional logic checks are implemented to ensure data consistency, including verifying whether the associations between different dimensions are correct and whether the calculated summary values match the underlying data;
[0118] E4. Recording and reporting: When data inconsistencies are found, relevant details are recorded immediately, including inconsistent data points, timestamps, cells involved, and detailed reports are generated;
[0119] E5. Problem tracing: trace back the source and processing of data to find the root cause of inconsistency;
[0120] E6. Data correction: Depending on the nature of the problem, correct the erroneous data or update the data processing logic to ensure future data consistency;
[0121] E7. Revalidation: Revalidate the corrected data to ensure that all issues have been resolved and the consistency and accuracy of the data have been restored.
[0122] The accuracy and reliability of data have been significantly improved through multiple verification methods such as source data verification, existing data verification in the dimension tree, and cross-cell consistency checks. The recording and reporting mechanism and problem tracing capabilities enable data problems to be discovered and effectively resolved in a timely manner. The closed-loop process of data correction and re-verification ensures the continuous restoration of data consistency and accuracy, providing users with higher-quality data assurance. These improvements have jointly improved the credibility of data processing and the overall performance of the system.
[0123] The operation method of the data filling module specifically includes:
[0124] F1. Fill the verified data into the corresponding cells of the dimension tree. The weighted average calculation method is used in the filling process. The calculation formula of the weighted average is: ;W m is the weight, X m is the value, m represents the mth data point;
[0125] Obtaining the weighted average has the following important implications for filling data into the corresponding cells of the dimension tree:
[0126] Improved accuracy: The weighted average takes into account the weights of different data points, which usually reflect the importance, reliability, or influence of the data points. Therefore, using the weighted average can more accurately reflect the overall characteristics of the data, especially when there are significant differences or unequal importance between data points;
[0127] Adapting to complex data structures: Dimension trees are often used to represent hierarchical data, where cells at different levels may contain data from different sources or with different importance. Weighted averages can flexibly handle such complex data structures, ensuring that the data populated in the dimension tree is both accurate and representative by assigning appropriate weights to different data points;
[0128] Support decision making: Dimension trees are often used in data analysis, reporting, and decision support systems. By calculating weighted averages and filling them into corresponding cells, decision makers can obtain more comprehensive and accurate information and make more informed decisions. The comprehensive perspective provided by weighted averages helps reveal trends, patterns, and relationships behind the data;
[0129] Optimizing performance: In some cases, the dimension tree may contain a large number of data points, and directly processing these data points may cause performance issues. By using a weighted average, multiple data points can be combined into a single value, reducing the complexity of data processing and the time required. This helps optimize the performance of the system and improve the efficiency of data processing;
[0130] Maintaining data consistency: When populating a dimension tree, it is very important to ensure data consistency and comparability. The weighted average provides a standardized method to handle data from different sources or under different conditions, so that the data populated in the dimension tree is logically consistent and facilitates cross-dimensional comparison and analysis.
[0131] F2. Implement update strategies, including direct coverage, incremental updates, or merged updates based on different business needs;
[0132] F3. During the filling process, use locking mechanisms or transaction processing to ensure data consistency when multiple threads or processes update the same cell;
[0133] F4. For updates of non-critical cells, an asynchronous strategy is used to improve the system's response speed and throughput;
[0134] F5. When performing performance testing, evaluate the processing time T PROCESS , THROUGHPUT indicators, which will be affected by the parameters and strategies in all previous steps.
[0135] In this embodiment, the data filling step includes:
[0136] F3.1. Locate the target position TP: Use the structural information of the dimension tree and the path information of the target filling position TP to locate the specific filling position in the dimension tree;
[0137] F3.2, Data formatting and filling: Format the pre-processed data according to the target format FG, and fill the formatted data into the target position TP;
[0138] F3.3, Verification and Feedback:
[0139] F3.3.1. Verify that the data is successfully filled into the target location and check the integrity and accuracy of the data;
[0140] F3.3.2. Feedback the filling results to the system administrator or user as needed, including success information, error logs or alarm notifications.
[0141] The weighted average calculation method and flexible update strategy were introduced to improve the accuracy and flexibility of data filling. At the same time, data consistency was ensured through locking mechanism or transaction processing to avoid conflicts during concurrent updates. The asynchronous update strategy optimized the update efficiency of non-critical cells and improved the system response speed and throughput. The introduction of performance testing made the system performance tuning more reliable and further improved the overall data processing capabilities and user experience. These improvements together enhanced the robustness and efficiency of the data filling module.
[0142] The operation method of the system monitoring and maintenance module specifically includes:
[0143] G1. Monitor system performance indicators, including CPU usage and memory usage. These indicators reflect the system's resource usage and potential performance bottlenecks.
[0144] G2. Adjust cache size according to monitoring data C SIZE 、Number of threads T THREADS , the number of shards DP parameters to optimize system performance. The adjusted parameters will directly affect key indicators such as cache hit rate (HR) and throughput (THROUGHPUT);
[0145] G3. Write automated test scripts to cover different data processing scenarios and perform system performance monitoring and parameter adjustment tests.
[0146] In this embodiment, the cache size C is adjusted SIZE 、Number of threads T THREADS The steps to adjust the parameters of the number of shards DP are:
[0147] G2.1. Set the CPU usage threshold to Y CPU , the memory usage threshold is Y MTU , and record CPU usage, memory usage, throughput, and cache hit rate indicators;
[0148] G2.2, Cache size C SIZE Adjustment:
[0149] If the cache hit rate C SIZE Low: Increase cache size C SIZE , to reduce access to backend storage;
[0150] If memory usage approaches or exceeds the threshold: Reduce cache size C SIZE ;
[0151] G2.3, number of threads T THREADS Adjustment:
[0152] If CPU usage is less than half of the threshold: increase the number of threads T THREADS , increase throughput;
[0153] If CPU usage approaches or exceeds the threshold: reduce the number of threads T THREADS ;
[0154] G2.4, Shard Quantity DP Adjustment:
[0155] If the system responds slowly when processing large data sets: increase the number of shards DP to distribute the processing pressure;
[0156] If the CPU and memory utilization of system resources are unbalanced: gradually adjust the number of shards DP until the resources are used in a balanced manner;
[0157] G2.5, Iterative adjustment and testing:
[0158] Repeat the test: After each parameter adjustment, rerun the performance test and record the new performance indicator data;
[0159] Evaluation effect: Compare the performance indicators before and after the adjustment to evaluate whether the adjustment is effective;
[0160] Continuous monitoring: After parameter adjustment, the system status is continuously monitored to ensure that the system operates stably within the threshold range.
[0161] By monitoring performance indicators in real time and automatically adjusting the cache, number of threads, and number of shards based on preset thresholds, the efficiency and accuracy of system performance tuning have been significantly improved. This automated adjustment mechanism can not only quickly respond to performance bottlenecks, but also find the best balance between resource utilization and performance, reduce manual intervention, and improve system stability and maintainability. At the same time, automated test scripts are introduced to ensure the effectiveness of adjustments, forming a closed-loop optimization process to continuously ensure efficient system operation.
[0162] Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.
[0163] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0164] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A centralized acquisition and filling system for data cells in a dimension tree, characterized in that: include: Data model and architecture, used to design the data model of the dimension tree and clarify the storage structure and access method of each cell; And plan the system architecture, design caching strategies to speed up the reading of frequently accessed cells, and determine the data sharding and partitioning scheme to distribute the access and update tasks of dimension tree cells; Data preprocessing module, used to clean the imported data source data and remove outliers; The parallel query and calculation module uses multi-threading or multi-process technology to query and obtain the data of dimension tree cells in parallel, and combines incremental aggregation and data compression rate algorithms to reduce the amount of calculation; Data validation module, which verifies the accuracy and legitimacy of data using checksums or hash values before populating the data into dimension tree cells; The data filling module fills the verified data into the corresponding cells of the dimension tree. The weighted average is used for calculation during the filling process, and the filling is performed according to the rules of the set update strategy. During the filling process, a lock mechanism or transaction processing is used to ensure the data consistency when multiple threads or processes update the same cell. For the update of non-critical cells, an asynchronous strategy is used to improve the system's response speed and throughput; The system monitoring and maintenance module monitors the system's performance indicators and flexibly adjusts the acquisition, processing and filling parameters of the dimension tree cell data to maintain the system performance indicators within the safety threshold.
2. The method for operating a system for centralized acquisition and filling of data cells in a dimension tree according to claim 1, characterized in that: Before data acquisition and filling, first perform the following setup steps: A1. Clarify the data source DS of the project and its characteristics, including data format, size and update frequency; A2. Determine the specific position of the target fill position TP in the dimension tree and the expected format FG; A3. Analyze the structure of the dimension tree and identify the number of nodes N and the number of levels L.
3. The method for operating a system for centralized acquisition and filling of data cells in a dimension tree according to claim 2, characterized in that: The specific methods of operating the data model and architecture include: B1. When designing the data model of the dimension tree, make sure each cell i,j The storage structure and access method of the node are as follows: i represents the row / level and j represents the column / attribute; and the node storage information is designed, including the node ID and the attribute set A; B2. When planning the system architecture, design a cache strategy to speed up the reading of frequently accessed cells, considering the cache size C SIZE With the expected data access pattern, the cache size C SIZE It will affect the cache hit rate HR. When the cache hit rate HR is improved, the number of accesses to the original data source will be reduced, thereby reducing the processing time T. P , the cache hit rate HR calculation formula is: ; Among them, T0 represents the number of hits, T 总 Indicates the total number of requests; B3. At the same time, determine the number of data shards DP and the partitioning strategy PART to evenly distribute the access and update tasks of the dimension tree cells during parallel processing.
4. The method for operating a system for centralized acquisition and filling of data cells in a dimension tree according to claim 2, characterized in that: The operation method of the data preprocessing module specifically includes: C1. Import the data source DS, extract the data related to the dimension tree cells, and obtain the clean data set DCS after cleaning. During the cleaning process, the threshold TO is used to determine and remove outliers.
5. The method for operating a system for centralized acquisition and filling of data cells in a dimension tree according to claim 2, characterized in that: The operation method of the parallel query and calculation module specifically includes: D1. Use multi-threading or multi-process technology to query and obtain the data of dimension tree cells from the data source or cache in parallel, and combine incremental aggregation IA technology for calculation optimization and data compression technology for compression processing; D2. In terms of computing optimization, incremental aggregation IA is used to reduce repeated computing. The calculation of incremental aggregation depends on the result of the previous state, that is: S new = S old + D; where Σ old represents the aggregate value before adding new data or changing; Δ represents the increment or change, that is, the change in the aggregate value caused by the addition of new data or the change of existing data; Σ new Represents the updated aggregate value by converting the old aggregate value Σ old Add to the increment Δ to get; The data compression ratio CR calculation formula is: ; Among them, C 前 The original data size before compression, C 后 Indicates the size of the compressed data.
6. The method for operating a system for centralized acquisition and filling of data cells in a dimension tree according to claim 2, characterized in that: The operation method of the data verification module specifically includes: E1. Source data verification: Before submitting data to the dimension tree, verify the checksum / hash value of the source data to ensure that the data has not been tampered with during transmission or preprocessing; E2. Verification of existing data in the dimension tree: If the dimension tree already contains some data, compare the checksum / hash value of the new data with the checksum / hash value of the stored data to check for potential duplication or conflict; E3. Cross-cell consistency check: For complex data relationships involving multiple cells, additional logic checks are implemented to ensure data consistency, including verifying whether the associations between different dimensions are correct and whether the calculated summary values match the underlying data; E4. Recording and reporting: When data inconsistencies are found, relevant details are recorded immediately, including inconsistent data points, timestamps, cells involved, and detailed reports are generated; E5. Problem tracing: trace back the source and processing of data to find the root cause of inconsistency; E6. Data correction: Depending on the nature of the problem, correct the erroneous data or update the data processing logic to ensure future data consistency; E7. Revalidation: Revalidate the corrected data to ensure that all issues have been resolved and the consistency and accuracy of the data have been restored.
7. The method for operating a system for centralized acquisition and filling of data cells in a dimension tree according to claim 2, characterized in that: The operation method of the data filling module specifically includes: F1. Fill the verified data into the corresponding cells of the dimension tree. The weighted average calculation method is used in the filling process. The calculation formula of the weighted average is: ; Where W m is the weight, X m is the value, m represents the mth data point; F2. Implement update strategies, including direct coverage, incremental updates, or merged updates based on different business needs; F3. During the filling process, use locking mechanisms or transaction processing to ensure data consistency when multiple threads or processes update the same cell; F4. For updates of non-critical cells, an asynchronous strategy is used to improve the system's response speed and throughput; F5. When performing performance testing, evaluate the processing time T PROCESS , throughput THROUGHPUT indicator.
8. The method for operating a system for centralized acquisition and filling of data cells in a dimension tree according to claim 7, characterized in that: The data filling steps include: F3.
1. Locate the target position TP: Use the structural information of the dimension tree and the path information of the target filling position TP to locate the specific filling position in the dimension tree; F3.2, Data formatting and filling: Format the pre-processed data according to the target format FG, and fill the formatted data into the target position TP; F3.3, Verification and Feedback: F3.3.
1. Verify that the data is successfully filled into the target location and check the integrity and accuracy of the data; F3.3.
2. Feedback the filling results to the system administrator or user as needed, including success information, error logs or alarm notifications.
9. The method for operating a system for centralized acquisition and filling of data cells in a dimension tree according to claim 2, characterized in that: The operation method of the system monitoring and maintenance module specifically includes: G1. Monitor system performance indicators, including CPU usage and memory usage; G2. Adjust cache size according to monitoring data C SIZE 、Number of threads T THREADS , the parameter of the number of shards DP; G3. Write automated test scripts to cover different data processing scenarios and perform system performance monitoring and parameter adjustment tests.
10. The method for operating a system for centralized acquisition and filling of data cells in a dimension tree according to claim 9, characterized in that: Adjust cache size C SIZE 、Number of threads T THREADS The steps to adjust the parameters of the number of shards DP are: G2.
1. Set the CPU usage threshold to Y CPU , the memory usage threshold is Y MTU , and record CPU usage, memory usage, throughput, and cache hit rate indicators; G2.2, Cache size C SIZE Adjustment: If the cache hit rate C SIZE Low: Increase cache size C SIZE , to reduce access to backend storage; If memory usage approaches or exceeds the threshold: Reduce cache size C SIZE ; G2.3, number of threads T THREADS Adjustment: If CPU usage is less than half of the threshold: increase the number of threads T THREADS , increase throughput; If CPU usage approaches or exceeds the threshold: reduce the number of threads T THREADS ; G2.4, Shard Quantity DP Adjustment: If the system responds slowly when processing large data sets: increase the number of shards DP to distribute the processing pressure; If the CPU and memory utilization of system resources are unbalanced: gradually adjust the number of shards DP until the resources are used in a balanced manner; G2.5, Iterative adjustment and testing: Repeat the test: After each parameter adjustment, rerun the performance test and record the new performance indicator data; Evaluation effect: Compare the performance indicators before and after the adjustment to evaluate whether the adjustment is effective; Continuous monitoring: After parameter adjustment, the system status is continuously monitored to ensure that the system operates stably within the threshold range.
