A spark-based data index dynamic calculation and broadening method and system

Abstract

The application discloses a kind of spark-based data index dynamic calculation and widening method and system;The present application relates to electric data processing technical field;Create view configuration information table, atomic index configuration information table and derived index configuration information table.Initialization configuration information table, including view name, job number, calculation logic, dependent relationship etc.Start Spark, load three configuration information tables on line to memory.According to the job name of incoming, query view configuration information table, obtain the construction logic and cache level of all views under this job.The present application realizes the flexible construction and calculation of view, atomic index and derived index by configuration management, reduces repeated coding work, shortens development cycle.Using the parallel computing capacity of big data processing framework such as Spark, optimize resource use, improve data processing speed.

Description

Technical Field

[0001] This invention relates to the field of electronic data processing technology, specifically to the layered construction of data warehouses and its DWS (Data Warehouse Service) layer, and particularly to a method and system for dynamic calculation and scaling of data metrics based on Spark. Background Technology

[0002] With the rapid development of the internet industry, various sectors are generating massive amounts of data in their daily operations, growing exponentially. This vast data resource not only contains enormous commercial value but also brings increasingly complex data analysis needs, leading to a sharp increase in the number of metrics that need to be analyzed and developed. Therefore, how to standardize the development, construction, management, and maintenance of metrics has become a crucial issue that urgently needs to be addressed.

[0003] In the layered construction of a data warehouse, the DWS (Data Warehouse Service) layer is a core component, and its table structure is mainly used to store metric data of the same dimension. However, the traditional approach is to integrate the calculated metrics into a single wide table. Therefore, during the development of data warehouse tasks, the calculation of metrics of the same dimension is usually coupled to the same task. This traditional approach means that if the calculation caliber of a metric is incorrect, or if the related task fails, all metrics in the entire table need to be recalculated. This high degree of coupling not only increases the complexity of maintenance but also reduces the scalability of the system.

[0004] Furthermore, the current indicator management system lacks effective launch and delaunch functions. This means that even if some indicators are no longer being monitored or used downstream, they will still be calculated repeatedly, resulting in serious waste of resources. Additionally, the current system does not clearly distinguish between atomic, derived, and supplementary indicators. This lack of classification limits the reusability of indicators and increases the complexity of developing new indicators.

[0005] To address this, the present invention proposes a method and system for dynamic calculation and broadening of data metrics based on Spark. Summary of the Invention

[0006] In view of this, the present invention aims to provide a method and system for dynamic calculation and broadening of data metrics based on Spark, in order to solve or alleviate the technical problems existing in the prior art, namely, how to implement a unified management mechanism for the development and maintenance processes of metrics based on configuration, and to at least provide a beneficial option for this; the technical solution of the present invention is implemented as follows:

[0007] Firstly, a method for dynamically calculating and broadening data metrics based on Spark:

[0008] (I) Overview:

[0009] This invention aims to achieve flexible construction and calculation of views, atomic metrics, and derived metrics through unified configuration information table management. The solution utilizes Spark as the computing engine, optimizing resource usage and improving data processing efficiency through parallel computing technology. First, the configuration information table is initialized and loaded into memory, and views are generated and cached according to job requirements. Second, atomic metric configuration information is read, dependent data is prepared, atomic metrics are calculated in parallel, and the results are stored. Next, the calculation logic for derived metrics is generated, the calculations are executed, and the results are stored in a Hive partitioned table. Finally, wide table query statements for online and offline metrics are generated, calculated, and the entire metric set is aggregated. This solution achieves unified management of metric development and maintenance processes, improves data processing and development efficiency, and provides enterprises with powerful data support capabilities.

[0010] (II) Technical Solution:

[0011] To achieve the above technical objectives, the present invention selects to perform the following steps.

[0012] 2.1 Step S1: Initialize the configuration information table, start the Spark program and load the configuration information:

[0013] Create and initialize the view configuration information table, atomic indicator configuration information table, and derived indicator configuration information table. The configuration information table includes the view name, job number, calculation logic, dependencies, etc.

[0014] Start Spark and load the three online configuration tables into memory. Based on the passed job name, query the view configuration table to obtain the build logic and cache level of all views under that job.

[0015] Simultaneously, views and caches need to be built: Views for this job should be generated and cached sequentially according to the configured view order and cache level. Ensure that view data is prepared and cached before calculating metrics.

[0016] 2.1.1 Step S100: Create a configuration information table:

[0017] Create a view configuration information table to store the name, job number, computation logic, and dependency configuration information of the view; create an atomic indicator configuration information table to store the computation logic and dependent view configuration information of the atomic indicator; create a derived indicator configuration information table to store the computation logic and atomic indicator configuration information of the derived indicator.

[0018] 2.1.2 Step S101, Initialize the configuration information table:

[0019] Insert initial view configuration data into the view configuration information table, including the view name, corresponding job number, computation logic description, and dependencies.

[0020] Insert initial atomic indicator configuration data into the atomic indicator configuration information table, including the calculation logic of the atomic indicator and the views it depends on.

[0021] Insert the initial derived indicator configuration data into the derived indicator configuration information table, including the calculation logic of the derived indicator and the atomic indicators it depends on.

[0022] 2.1.3 Step S102, Start the Spark program:

[0023] Initialize the Spark environment and configure parameters, including memory size and the number of executors. Use the Spark program to connect to the database or read the configuration file, loading the contents of the view configuration table, atomic metric configuration table, and derived metric configuration table into the Spark program's memory.

[0024] 2.1.4 Step S103, query view configuration information:

[0025] Based on the input job name, query the view configuration information table to obtain the build logic and cache level of all views under that job. Store the query results in the Spark program's memory so that views can be generated and cached subsequently according to the configured view order and cache level.

[0026] 2.1.5 Step S104, Build the view and cache:

[0027] Based on the view order and caching level in the query results, the Spark program is used to generate data for each view in sequence and cache the generated view data in the Spark program so that the cached view data can be used directly when calculating metrics, thereby improving processing efficiency.

[0028] 2.2 Step S2, Obtain the atomic index calculation logic:

[0029] Read the atomic metric configuration information table to identify the calculation logic set for all online atomic metrics. Determine the view that each atomic metric depends on and prepare the corresponding data.

[0030] 2.2.1 Step S200: Read the atomic index configuration information table:

[0031] Use a Spark program to connect to the database, execute SQL queries, and read the configuration information of all online atomic metrics from the atomic metric configuration information table. Store the query results in the Spark program's memory for later processing.

[0032] 2.2.2 Step S201, extract the set of atomic index calculation logic:

[0033] The atomic indicator configuration information in memory is traversed, and the calculation logic for each atomic indicator is extracted. The calculation logic is then organized according to the name of the atomic indicator to form a set of calculation logic.

[0034] 2.2.3 Step S202, determine the view on which the atomic index depends:

[0035] For each atomic metric, analyze its calculation logic and determine the views it depends on. Associate and store the names of the dependent views with their corresponding atomic metric names for later data preparation.

[0036] 2.2.4 Step S203, prepare the relevant data:

[0037] Based on the view name that the atomic metric depends on, the corresponding view data is retrieved from the Spark program's cache. If the required view data exists in the cache, proceed to S3; otherwise, the data is regenerated according to the view's construction logic and cached. This ensures that all view data dependent on atomic metrics are ready and up-to-date.

[0038] 2.3 Step S3: Parallel computation of atomic indices and storage of results:

[0039] Asynchronous multithreading is used to trigger Spark tasks in parallel, simultaneously calculating multiple atomic metrics. The number of metrics calculated in each parallel iteration is controlled by a configured step size parameter to optimize resource usage. The calculation results of all atomic metrics are then written to a Hive three-level partitioned table.

[0040] 2.3.1 Step S300: Asynchronously trigger Spark tasks in parallel using multiple threads:

[0041] Leveraging Spark's parallel computing capabilities, multiple Spark tasks are triggered asynchronously and in a multi-threaded manner. Each task is responsible for calculating one or more atomic metrics. Spark's task scheduling mechanism ensures concurrent execution of tasks and efficient utilization of resources.

[0042] 2.3.2 Step S301: Control the number of metrics for parallel computing:

[0043] The number of atomic metrics computed in each parallel operation is controlled by the configured step size parameter. This step size parameter can be adjusted based on system resources and computational needs to optimize resource utilization and computational efficiency. Within each Spark task, computation is performed simultaneously by partitioning, transforming, and aggregating the data based on the computation logic of the atomic metrics and the dependent view data, utilizing Spark's distributed computing framework.

[0044] 2.3.3 Step S302, store the calculation results:

[0045] The calculation results for each atomic metric are written to a three-level partition table in Hive. The partition table includes partition information at three levels: date, business line name, and metric name, facilitating subsequent data querying and analysis. Hive's insert operation is used to store the calculation results according to the partition information.

[0046] 2.4 Step S4: Generation, calculation, and storage of derived indicator calculation logic:

[0047] Read the atomic metric configuration information table and the derived metric configuration information table to find the set of atomic and derived metrics corresponding to the job. Generate an atomic metric view for the job as a subquery for calculating derived metrics. Simultaneously, based on the metadata fields and order of the wide table, traverse and generate the calculation logic for derived metrics, execute the calculation of derived metrics, and store the results in the corresponding Hive partition table.

[0048] 2.4.1 Step S400: Read the configuration information table:

[0049] Use a Spark program to connect to the database and read the atomic indicator configuration information table and the derived indicator configuration information table respectively.

[0050] From the atomic metric configuration information table, filter out the set of atomic metrics corresponding to this job.

[0051] From the derived metric configuration information table, filter out the set of derived metrics corresponding to this job.

[0052] 2.4.2 Step S401: Generate atomic index view:

[0053] Based on the selected set of atomic metrics, generate an atomic metric view for the job. Iterate through the selected set of derived metrics and process each derived metric.

[0054] 2.4.3 Step S402: Generate the calculation logic for derived indicators:

[0055] For each derived metric, a corresponding calculation logic is generated based on the calculation logic in its configuration information and the atomic metrics it depends on. This calculation logic will use the previously generated atomic metric view as a subquery.

[0056] 2.4.4 Step S403: Calculate the derived indicator:

[0057] Based on the generated derived metric calculation logic, perform calculation operations. During the calculation process, Spark's distributed computing capabilities can be utilized to process the data.

[0058] Finally, the calculation results of each derived indicator are written into the corresponding Hive partition table.

[0059] Step 2.5, S5: Generation and summarization of wide tables and offline metrics:

[0060] The Spark program performs calculations to generate online metric wide tables and offline metric query statements. The calculation results are then written to the online summary layer metric wide table.

[0061] Generate a wide table of offline summary layer metrics. Perform a full join operation on the online and offline wide tables to generate a full set of metrics. Then, write the full set of metrics into the final metric summary table.

[0062] 2.5.1 Step S500: Generate a wide table of online metrics.

[0063] Using a Spark program, a wide table of online metrics is created based on the previously obtained atomic and derived metrics. This wide table will contain multiple metrics, each serving as a column of the wide table.

[0064] Then, use Hive's insert operation to write the data from the generated online metric wide table into the online summary layer metric wide table.

[0065] Generate a wide table of metrics for the offline summary layer based on business needs. This table will contain the summary metrics required for offline business operations.

[0066] 2.5.2 Step S501: Perform a full connection operation.

[0067] Perform a full join operation on the online summary layer indicator wide table and the offline summary layer indicator wide table. This ensures that all indicators in both wide tables are included in the full indicator set.

[0068] Through fully connected operations, a complete set of metrics, including all online and offline indicators, is generated. This set will serve as the final data output.

[0069] 2.5.3 Step S502: Write the data into the final indicator summary table:

[0070] Using Hive's insert operation, the entire set of metrics is written into the final metric summary table, ensuring data persistence and queryability.

[0071] (III) Mechanisms for solving technical problems:

[0072] 3.1 Parameterization and Externalization of Configuration Information:

[0073] 3.1.1 Parameterized Configuration:

[0074] The behavior and characteristics of metrics can be flexibly configured through configuration files, environment variables, or command-line arguments. This design allows for adjustments to the calculation logic and dependencies of metrics based on different needs and environments without modifying the source code.

[0075] 3.1.2 Externalized Configuration:

[0076] Separate the metric configuration information from the code and store it in a separate configuration file or database. This simplifies the configuration modification and management process, and improves configuration transparency and maintainability.

[0077] 3.2 Construction of the Configuration Management System:

[0078] 3.2.1 Design of the configuration information table:

[0079] Design and create view configuration information tables, atomic indicator configuration information tables, and derived indicator configuration information tables to store detailed information of various configuration items, including names, calculation logic, dependencies, etc.

[0080] 3.2.2 Loading and caching of configuration information:

[0081] At the start of the data processing flow, these configuration information tables are loaded into memory by the Spark program, and the corresponding configuration information is queried and cached according to job requirements. This ensures that the configuration information can be quickly accessed and referenced during processing.

[0082] 3.3 Automated Data Processing and Indicator Calculation:

[0083] 3.3.1 View Construction and Caching:

[0084] Based on the view building logic and caching level in the configuration information table, a Spark program sequentially generates and caches views. These views serve as the data source for subsequent metric calculations, ensuring data consistency and availability.

[0085] 3.3.2 Calculation of Atomic Indices and Derivative Indices:

[0086] The system reads the atomic metric configuration information table to determine the calculation logic and dependencies of each atomic metric, and utilizes Spark's parallel computing capabilities to calculate multiple atomic metrics simultaneously. Then, it generates the calculation logic for derived metrics based on the derived metric configuration information table, and calculates the derived metrics based on the calculation results of the atomic metrics.

[0087] 3.3.3 Result Storage and Summarization:

[0088] The calculation results of atomic and derived metrics are stored in a Hive partitioned table for easy subsequent querying and analysis. Simultaneously, wide tables for online and offline metrics are generated according to requirements, and these wide tables are aggregated to form a complete set of metrics.

[0089] 3.4 Implementation of a unified management mechanism

[0090] 3.4.1 Configuration Change Management:

[0091] When configuration information changes (such as adding metrics or modifying calculation logic), only the contents of the configuration information table need to be updated; no code modification or application redeployment is required. Automated tools periodically synchronize changes to the configuration information table into the data processing workflow, ensuring the real-time nature and consistency of configuration information.

[0092] 3.4.2 Performance Optimization and Monitoring:

[0093] The number of metrics computed in parallel is controlled by configuring step size parameters to optimize resource utilization and improve processing efficiency. Simultaneously, a monitoring mechanism is established to monitor the performance and status of the data processing flow in real time, ensuring stable system operation and timely handling of anomalies.

[0094] Secondly, a Spark-based system for dynamically calculating and scaling data metrics:

[0095] The system includes a processor and a memory connected to the processor. The memory stores program instructions, which, when executed by the processor, cause the processor to perform the dynamic calculation and broadening method for data indicators as described above. The processor is also connected to:

[0096] (1) Initialize the configuration information table, start the Spark program, and load the Spark program module with the configuration information;

[0097] (2) Atomic indicator calculation logic module for extracting atomic indicators and preparing the view data on which the atomic indicators depend.

[0098] (3) Parallel computing module for parallel computing of multiple atomic indicators and storing the computing results in a Hive partition table;

[0099] (4) The calculation logic for generating derived indicators, and the derived indicator calculation logic module responsible for calculating derived indicators and storing the results in the Hive partition table.

[0100] (5) Used to generate wide tables of online and offline indicators, perform full connection operations on the wide tables to generate a full set of indicators, and store them in the generation and summary module of the final indicator summary table.

[0101] Compared with the prior art, the beneficial effects of the present invention are:

[0102] I. Improved Efficiency of Data Processing and Metric Development: This invention, through configuration-based management, enables flexible construction and calculation of views, atomic metrics, and derived metrics, reducing repetitive coding work and shortening the development cycle. Leveraging the parallel computing capabilities of big data processing frameworks such as Spark optimizes resource utilization and improves data processing speed. Furthermore, the parameterized and externalized management of configuration information means that adding or modifying metrics requires only updating configuration files or the database, without modifying the code, thus reducing maintenance costs. Simultaneously, the unified configuration management mechanism makes the metric development and maintenance processes more standardized and regulated, improving the system's scalability.

[0103] Second, it improves data quality and accuracy: This invention reduces human error and intervention through automated data processing and indicator calculation, thereby improving data accuracy and reliability. Real-time synchronization of configuration information changes ensures the real-time nature and consistency of data processing logic.

[0104] III. Optimized Data Support Capabilities: This invention facilitates data querying and analysis through the construction of wide tables and summary layers, supporting data-driven decision-making for enterprises. Furthermore, by configuring step size parameters and other methods, the number of metrics computed in parallel is controlled, resource utilization is optimized, and system performance is improved. Attached Figure Description

[0105] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0106] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0107] Figure 2 This is a schematic diagram of the system composition of the present invention;

[0108] Figure 3 A schematic diagram of the architecture of the view configuration information table of this invention;

[0109] Figure 4 A flowchart illustrating the architecture of the atomic index configuration information table for this invention;

[0110] Figure 5 The diagram shows the architecture of the information table for the derived indicators of this invention. Detailed Implementation

[0111] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below;

[0112] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0113] Glossary:

[0114] (1) Spark: A fast and versatile computing engine for large-scale data processing and analysis.

[0115] (2) Hive: A data warehouse tool based on Hadoop for extracting, transforming and loading data.

[0116] (3) MySQL: A relational database management system.

[0117] (4) Partitioning: Data in a data warehouse is categorized and stored physically, reducing the amount of data scanning. Typical partitions are based on date and can be divided into second- or third-level partitions. Data in the specified partition can be queried as needed.

[0118] (5) Atomic indicators: Indicators that cannot be split.

[0119] (6) Derived indicators: atomic indicators + statistical period + business + statistical granularity.

[0120] (7) Derived indicators: These are derived from one or more derived indicators through various logical operations.

[0121] (8) Online metrics: Metrics that are being used by the business system and have a real impact.

[0122] (9) Offline metrics: Metrics that are no longer being monitored and have no practical use downstream. They are mainly used for testing and verifying the rationality of the data metrics.

[0123] Example 1: As Figure 1 As shown, this embodiment provides a method for dynamically calculating and broadening ride-hailing customer data using Spark-based data metrics. The method includes the following steps:

[0124] In this embodiment, regarding step S1: configuration information table initialization and Spark program startup:

[0125] Specifically, create a configuration information table (S100):

[0126] View configuration information table, such as Figure 3 As shown: This table stores view information related to ride-hailing customer data processing, including view name, job number, calculation logic, and dependencies. Views are an important part of the data processing process; they typically represent an intermediate state or result of the data processing.

[0127] Atomic indicator configuration information table, such as Figure 4 As shown: This stores the calculation logic of atomic metrics and the views they depend on. Atomic metrics are indivisible metrics, such as "order volume" and "passenger count," and they form the basis for more complex metrics (such as derived metrics).

[0128] Derivative indicator configuration information table, such as Figure 5 As shown: This demonstrates the calculation logic of derived metrics and the atomic metrics they depend on. Derived metrics are derived from atomic metrics through specific calculation logic, such as "average order amount" and "passenger growth rate".

[0129] Specifically, the initialization configuration information table (S101) inserts view configuration data related to ride-hailing customer data processing into the view configuration information table, such as "Customer Order View" and "Customer Trip View". It also inserts atomic indicator configuration data related to ride-hailing customers into the atomic indicator configuration information table, such as the calculation logic of the "Order Volume" indicator and the views it depends on.

[0130] Insert the configuration data of derivative indicators related to ride-hailing customers into the derivative indicator configuration information table, such as the calculation logic of the "average order amount" indicator and the atomic indicators it depends on.

[0131] Specifically, starting the Spark program (S102): Initializes the Spark environment, configuring appropriate memory size and the number of executors based on the scale of ride-hailing customer data processing. Connects to the database or reads the configuration file, loading the contents of the view configuration table, atomic metric configuration table, and derived metric configuration table into the Spark program's memory for subsequent fast access and processing.

[0132] Specifically, query view configuration information (S103): Based on the input job name (e.g., "Ride-hailing Customer Daily Report"), query the view configuration information table to obtain the construction logic and cache level of all views under that job. This step ensures that subsequent data processing can proceed according to the predetermined logic and order.

[0133] Specifically, the view construction and caching (S104) process generates data for each view sequentially according to the view order and cache level in the query results. For example, first generate the "Customer Orders View," then generate the "Customer Itinerary View." The generated view data is cached in the Spark program so that it can be directly used when calculating metrics, improving processing efficiency.

[0134] In this embodiment, regarding step S2: obtaining the atomic index calculation logic:

[0135] Specifically, the atomic metric configuration information table (S200) is read: A Spark program connects to the database and reads the configuration information for all online atomic metrics. This information includes the calculation logic of the atomic metrics and the views they depend on.

[0136] Specifically, extract the set of atomic indicator calculation logic (S201): traverse the atomic indicator configuration information in memory and extract the calculation logic for each atomic indicator. For example, extract the calculation logic for the "order volume" indicator.

[0137] Specifically, determine the views that the atomic metrics depend on (S202): Analyze the calculation logic of each atomic metric to determine the views it depends on. For example, the "order volume" metric may depend on the "customer order view".

[0138] Specifically, prepare the relevant data (S203): Look up the view data that the atomic metrics depend on from the Spark program's cache. If the required view data exists in the cache, it is used directly; otherwise, the data is regenerated according to the view's construction logic and cached. This step ensures that all view data that the atomic metrics depend on is ready and up-to-date.

[0139] In this embodiment, regarding step S3: parallel computation of atomic indices and storage of results:

[0140] Specifically, asynchronous multi-threaded parallel triggering of Spark tasks (S300): Utilizing Spark's parallel computing capabilities, multiple Spark tasks are triggered asynchronously and in a multi-threaded manner. Each task is responsible for calculating one or more atomic metrics, such as "order volume" or "passenger count".

[0141] Specifically, controlling the number of metrics in parallel computation (S301): Based on the configured step size parameter (e.g., calculating 5 metrics each time), the number of atomic metrics in each parallel computation is controlled. This step is to optimize resource utilization and computational efficiency, preventing resource contention and performance degradation due to too many parallel tasks.

[0142] Specifically, storing the calculation results (S302): The calculation result of each atomic indicator is written to a Hive three-level partition table. The partition table includes partition information at three levels: date, business line name, and indicator name, which facilitates subsequent data querying and analysis.

[0143] In this embodiment, regarding step S4: the generation, calculation, and storage of derived index calculation logic:

[0144] Specifically, the configuration information table (S400) is read: A Spark program connects to the database and reads the atomic metric configuration information table and the derived metric configuration information table. This step is to obtain the calculation logic of the derived metrics and the information of the atomic metrics they depend on.

[0145] Specifically, generate atomic metric views (S401): Based on the selected set of atomic metrics (such as "order volume", "passenger count", etc.), generate atomic metric views for this job. These views will be used as subqueries for calculating derived metrics.

[0146] Specifically, the calculation logic for generating derived metrics (S402) is generated: For each derived metric (such as "average order amount", "passenger growth rate", etc.), the corresponding calculation logic is generated based on the calculation logic in its configuration information and the atomic metrics it depends on.

[0147] Specifically, the derived metric calculation (S403) is performed: Based on the generated derived metric calculation logic, calculation operations are executed. During the calculation process, Spark's distributed computing capabilities can be used to process the data. Finally, the calculation results of each derived metric are written to the corresponding Hive partition table.

[0148] In this embodiment, regarding step S5: the generation and summarization of wide tables and offline metrics:

[0149] Specifically, the online metrics wide table (S500) is generated: using a Spark program, a wide table of online metrics is formed based on the previously obtained atomic and derived metrics. This wide table will contain multiple metrics, each as a column of the wide table. Then, Hive's insert operation is used to write the data from the generated online metrics wide table into the online summary layer metrics wide table.

[0150] Specifically, a full join operation (S501) is performed: a full join operation is performed on the online summary layer indicator wide table and the offline summary layer indicator wide table. This step is to ensure that all indicators in both wide tables are included in the full indicator set, facilitating subsequent data analysis and decision-making.

[0151] Specifically, the data is written to the final indicator summary table (S502): Using Hive's insert operation, the entire set of indicators is written to the final indicator summary table. This step ensures data persistence and queryability, facilitating subsequent data analysis and mining.

[0152] Example 2: Figure 2 As shown, this embodiment, based on embodiment one, further discloses a data metric dynamic calculation and broadening system based on Spark:

[0153] The system includes a processor and a memory connected to the processor. The memory stores program instructions, which, when executed by the processor, cause the processor to perform the dynamic calculation and broadening method for data indicators as described above. The processor is also connected to:

[0154] (1) Initialize the configuration information table, start the Spark program, and load the Spark program module with the configuration information;

[0155] Main steps:

[0156] Design and create three configuration information tables: view configuration, atomic indicator configuration, and derived indicator configuration.

[0157] Insert the initial configuration data into these three tables.

[0158] Start the Spark program and perform the necessary configurations.

[0159] The contents of three configuration information tables are read from the database and loaded into Spark's memory.

[0160] Based on the input job name, query the view configuration information table to obtain the relevant view's construction logic and caching level.

[0161] Generate and cache view data according to the configured view order and cache level to ensure that the view data is ready before calculating metrics.

[0162] (2) Atomic indicator calculation logic module for extracting atomic indicators and preparing the view data on which the atomic indicators depend.

[0163] Main steps:

[0164] Read the configuration of all online atomic metrics from the atomic metric configuration information table.

[0165] Extract the computational logic for each atomic indicator and determine the views it depends on.

[0166] Prepare the corresponding view data based on the dependencies.

[0167] (3) Parallel computing module for parallel computing of multiple atomic indicators and storing the computing results in a Hive partition table;

[0168] Main steps:

[0169] Leveraging Spark's parallel computing capabilities, multiple tasks are triggered asynchronously and in a multi-threaded manner, with each task responsible for calculating one or more atomic metrics.

[0170] The number of metrics computed in each parallel operation can be controlled by configuring the step size parameter, thereby optimizing resource utilization.

[0171] Write the calculation results of all atomic metrics into a Hive three-level partition table, including the date, business line name, and metric name as partition information.

[0172] (4) The calculation logic for generating derived indicators, and the derived indicator calculation logic module responsible for calculating derived indicators and storing the results in the Hive partition table.

[0173] Main steps:

[0174] Read the configuration information of atomic and derived metrics to find the set of atomic and derived metrics for the corresponding job.

[0175] Generate an atomic indicator view as a subquery for calculating derived indicators.

[0176] Iterate through the set of derived indicators and generate the calculation logic for each derived indicator.

[0177] Perform the calculation of derived metrics and store the results in the corresponding Hive partition table.

[0178] (5) The generation and summary module is used to generate wide tables of online and offline indicators, perform full connection operations on the wide tables, generate a full set of indicators, and store them in the final indicator summary table.

[0179] Main steps:

[0180] Use Spark programs to calculate and generate a wide table of online metrics.

[0181] Write the data from the online indicator wide table into the online summary layer indicator wide table.

[0182] Generate a wide table of metrics for the offline summary layer.

[0183] Perform a full join operation on the online and offline wide tables to generate a complete set of metrics.

[0184] Write the complete set of indicators into the final indicator summary table for subsequent data querying and analysis.

[0185] All the above embodiments merely illustrate implementation methods for relevant practical applications of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

[0186] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0187] Furthermore, those skilled in the art will understand that implementing all or part of the processes in all the above-described embodiments can be accomplished by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media provided in this application and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

Claims

1. A method for dynamically calculating and broadening data metrics based on Spark, characterized in that, Includes the following steps: S1. Create and initialize the view configuration information table, atomic indicator configuration information table, and derived indicator configuration information table. Start Spark and load the view configuration information table, atomic indicator configuration information table, and derived indicator configuration information table into memory. Based on the passed job name, query the view configuration information table to obtain the construction logic and cache level of all views under that job; Based on the view order and cache level in the query results, the Spark program is used to generate data for each view in sequence and the generated view data is cached in the Spark program. S2, read the atomic indicator configuration information table, find the calculation logic set of all online atomic indicators; determine the view that each atomic indicator depends on; Based on the view name that the atomic metric depends on, look up the corresponding view data in the Spark program's cache; if the required view data exists in the cache, proceed to S3; Conversely, if the data is not generated according to the view's construction logic, it will be cached. S3 utilizes asynchronous multithreading to trigger Spark tasks in parallel, simultaneously calculating multiple atomic metrics. Then, the calculation results of all atomic indicators are written into the Hive three-level partition table; The implementation method of S3 includes: Asynchronously trigger multiple Spark tasks using multiple threads, with each task responsible for calculating one or more atomic metrics; Based on the configured step size parameter, the number of atomic metrics calculated in each parallel operation is controlled. In each Spark task, based on the calculation logic of the atomic metrics and the dependent view data, Spark's distributed computing framework is used to perform calculations while partitioning, transforming and aggregating the data. The calculation results of each atomic metric are written into a Hive three-level partition table; the partition table includes partition information at three levels: date, business line name, and metric name; S4: Read the atomic indicator configuration information table and the derived indicator configuration information table, find the set of atomic indicators and derived indicators corresponding to the job; generate the atomic indicator view of the job as a subquery, and at the same time, according to the wide table metadata information fields and order, traverse and generate the calculation logic of the derived indicators, execute the calculation of the derived indicators, and store the results in the corresponding Hive partition table. S5 uses Spark programs to calculate and generate online indicator wide tables and query statements for offline indicators. It generates offline summary layer indicator wide tables, performs full join operations, generates a full set of indicators, and writes them into the final indicator summary table.

2. The method for dynamically calculating and broadening data indicators according to claim 1, characterized in that: The implementation method of S1 includes: S100, create a view configuration information table to store the name, job number, computation logic, and dependency configuration information of the view; create an atomic indicator configuration information table to store the computation logic and dependent view configuration information of the atomic indicator; create a derived indicator configuration information table to store the computation logic and atomic indicator configuration information of the derived indicator. S101, Insert initial view configuration data into the view configuration information table, including the view name, corresponding job number, computation logic description and dependencies; Insert initial atomic metric configuration data into the atomic metric configuration information table, including the calculation logic of the atomic metrics and the views they depend on; Insert initial derived indicator configuration data into the derived indicator configuration information table, including the calculation logic of the derived indicator and the atomic indicators it depends on; S102 initializes the Spark environment and configures parameters, including memory size and number of executors; uses the Spark program to connect to the database or read configuration files, loading the contents of the view configuration information table, atomic metric configuration information table, and derived metric configuration information table into the Spark program's memory.

3. The method for dynamically calculating and broadening data indicators according to claim 1, characterized in that: The implementation method of S2 also includes: S200 uses a Spark program to connect to the database, executes SQL queries, and reads the configuration information of all online atomic metrics from the atomic metric configuration information table. S201, traverse the atomic indicator configuration information in memory, extract the calculation logic of each atomic indicator; organize the calculation logic according to the name of the atomic indicator to form a set of calculation logic; S202, for each atomic index, analyze its calculation logic and determine the view it depends on; associate and store the dependent view name and the corresponding atomic index name.

4. The method for dynamically calculating and broadening data indicators according to claim 1, characterized in that: The implementation method of S4 includes: S400 uses a Spark program to connect to the database and read the atomic indicator configuration information table and the derived indicator configuration information table respectively. Filter the set of atomic metrics corresponding to the job from the atomic metric configuration information table; From the derived metric configuration information table, filter out the set of derived metrics corresponding to this job; S401: Generate an atomic metric view for the job based on the selected set of atomic metrics; iterate through the selected set of derived metrics and process each derived metric. S402, For each derived metric, generate the corresponding calculation logic based on the calculation logic in its configuration information and the atomic metrics it depends on; S403, based on the generated derivative indicator calculation logic, performs calculation operations and writes the calculation results of each derivative indicator into the corresponding Hive partition table.

5. The method for dynamically calculating and broadening data indicators according to claim 1, characterized in that: The implementation method of S5 includes: S500 generates a wide table of online metrics based on previously obtained atomic and derived metrics, and uses Hive's insert operation to write the generated wide table of online metrics into the wide table of online summary layer metrics; it also generates a wide table of offline summary layer metrics according to business needs. S501 performs a full join operation on the online summary layer indicator wide table and the offline summary layer indicator wide table; allowing all indicators in both wide tables to be included in the full indicator set; through the full join operation, a full indicator set containing all online and offline indicators is generated. S502 uses Hive's insert operation to write the full set of metrics into the final metric summary table.

6. A system for dynamically calculating and broadening data metrics based on Spark, characterized in that: The system includes a processor and a memory connected to the processor. The memory stores program instructions, which, when executed by the processor, cause the processor to perform the data indicator dynamic calculation and broadening method as described in any one of claims 1-5.

Images