A data indicator management system

Abstract

The present invention discloses a data indicator management system, and relates to the field of data processing technology. In the data indicator management system provided by the present invention, an indicator management platform creates indicators and stores the created indicators in a platform database to realize the maintenance configuration of the platform database; an indicator driving engine application monitors the indicators stored in the platform database, and triggers the indicator data processing flow according to the maintenance configuration of the indicator management platform in the platform database; a data processing related component implanted with an indicator data processing application initiates a data request to a target data source, and processes the acquired indicator data according to the configuration to obtain indicator information; an indicator integration platform integrates the indicator information according to set conditions to generate report information, thereby being able to reduce the operation and maintenance cost and data consistency solution requirements, and providing a one-stop solution for simple scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data indicator management system. Background Art

[0002] Data warehouse (DW or DWH) is a strategic collection of all types of data support for decision-making processes at all levels of the enterprise. It is a single data store. Created for analytical reporting and decision support purposes, it provides guidance for business process improvement, monitoring time, cost, quality, and control for enterprises that need business intelligence.

[0003] In the IT departments of Internet organizations or financial institutions, data warehouses (abbreviated as data warehouses) are implemented. Data is stored in data warehouses in different categories. Applications customize development logic to obtain data warehouse resources according to their own needs.

[0004] Data analysis and decision-making need to rely on quantitative indicators, which in turn come from different business scenarios, data sources, and calculation calibers. Therefore, a set of methods and tools are needed to standardize the definition, development, and maintenance of indicators. As a data indicator management system, the indicator center can decouple and split the components of indicators and define them in a standardized manner in a logical table. On this basis, they can be freely assembled according to certain rules to realize the function of custom indicators. Storing and extracting indicators through metadata centers, data warehouses, or data lakes can improve data availability and response speed, while supporting data visualization and analysis.

[0005] The traditional concept of data warehouse is to use ETL tools to filter and clean data, integrate scattered data, and store the data in the data warehouse. Figure 1 shown.

[0006] Although the existing solution improves the traditional ETL data warehouse concept, the main purpose is to standardize indicators and combine them with business attributes, and can integrate company-level data into one system to facilitate the use of non-professionals, but the existing indicator center has the following problems:

[0007] 1) Data sources are mainly acquired through SQL, and there is a lack of other forms of data collection solutions;

[0008] 2) The ETL production process has a high threshold and lacks mature low-code solutions;

[0009] 3) The system performance is relatively low and cannot withstand large-scale access;

[0010] 4) It only saves data sets and lacks modeling and report display functions. Summary of the invention

[0011] In order to solve the above problems existing in the prior art, the present invention provides a data indicator management system.

[0012] To achieve the above object, the present invention provides the following solutions:

[0013] A data indicator management system, comprising:

[0014] The indicator management platform is used to create indicators and store the created indicators in the platform database to implement the maintenance and configuration of the platform database;

[0015] An indicator-driven engine application is used to monitor the indicators stored in the platform database and trigger the indicator data processing flow according to the maintenance configuration of the indicator management platform in the platform database;

[0016] Data processing related components are embedded with indicator data processing applications, which are used to initiate data requests to the target data source and process the obtained indicator data according to the configuration to obtain indicator information;

[0017] The indicator integration platform is used to integrate the indicator information according to set conditions to generate report information.

[0018] Optionally, the indicator management platform uses an indicator management page to create indicators; the editing information performed during the indicator creation process includes data source information, trigger information and data processing flow information.

[0019] Optionally, when the indicator-driven engine application monitors the platform database data, when a trigger condition is met, the indicator processing flow is automatically called, and a local or third-party call is made through an API.

[0020] Optionally, the indicator-driven engine application uses the automatic task component xxl-job to trigger the indicator processing flow and selects an execution mode during the indicator processing flow; the execution mode includes a dependent execution mode and an independent execution mode.

[0021] Optionally, the indicator data processing application includes:

[0022] Multi-source data collection application, used to obtain source data from the outside according to the configured data source information; in the process of obtaining source data, other indicator data are called as variables through the configuration information saved in the indicator management platform to realize the dynamic demand of source data;

[0023] Low-code platform applications have fixed processing logic in the data processing process. The processing logic is used to process data, and after data processing, Sink is used to store the processed data in Redis for further processing; the fixed processing logic includes: row-to-column, column-to-row, and splitting a single column into multiple rows.

[0024] Multilingual data processing applications, used to index data by writing code during the data processing process;

[0025] Data modeling applications are used to structure indicators.

[0026] Optionally, the functions integrated by the multi-source data collection application include: reading databases, APIs, message queues and files; wherein the database is acquired using JDBC, and the API is processed using components corresponding to the API.

[0027] Optionally, the multi-language data processing application uses graalvm as the JVM to support multiple programming languages ​​running in the JVM; in the multi-language data processing application, an alternative solution for script processing is used using the Python interpreter, and the execution results are stored in Redis; and the execution results saved in Redis are used when writing scripts.

[0028] Optionally, the structured implementation process of the indicator is: the processed indicator result set is processed using an algorithm to obtain a DDL data; when executed for the first time, the data modeling application is used to create a table in the data warehouse to achieve the purpose of determining the data dimension in the data warehouse based on the configuration of the indicator management platform.

[0029] Optionally, the indicator integration platform customizes and displays the generated report information according to customer needs.

[0030] Optionally, the indicator integration platform is also used to obtain indicator results using Redis data.

[0031] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0032] In the data indicator management system provided by the present invention, the indicator management platform creates indicators and stores the created indicators in the platform database to realize the maintenance configuration of the platform database; the indicator driving engine application monitors the indicators stored in the monitoring platform database, and triggers the indicator data processing flow according to the maintenance configuration of the indicator management platform in the platform database; the data processing related components implanted with the indicator data processing application initiate data requests to the target data source, and process the acquired indicator data according to the configuration to obtain indicator information; the indicator integration platform integrates the indicator information according to the set conditions to generate report information, thereby being able to reduce the operation and maintenance costs and data consistency solution requirements, and provide a one-stop solution for simple scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 This is a schematic diagram of the existing data warehouse;

[0035] Figure 2 A schematic diagram of the data indicator management system provided by the present invention;

[0036] Figure 3 This is a flowchart of the indicator processing provided by the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] The purpose of the present invention is to provide a data indicator management system with low operation and maintenance costs, which can reduce the solution requirements for data consistency and provide a one-stop solution for simple scenarios.

[0039] Description of prior art terms:

[0040] The ETL tool is a process that extracts, cleans, and converts data from the business system and then loads it into the data warehouse. The purpose is to integrate the scattered, disorganized, and non-uniform data in the enterprise to provide an analytical basis for the enterprise's decision-making.

[0041] Typescript is an open source programming language developed by Microsoft, built by adding static type definitions to JavaScript. Typescript is translated into JavaScript code through the TypeScript compiler or Babel, and can run in any browser and any operating system. Typescript supports header files that add type information to existing JavaScript libraries, expanding its support for popular libraries such as jQuery, MongoDB, Node.js, and D3.js. The type definitions of these third-party libraries are also open source, and all developers can contribute.

[0042] Vue3 development framework: an open source technology development framework, which is a popular front-end technology language with an active community on the market.

[0043] Java is a programming language that is specifically designed for the distributed environment of the Internet. Java has a "form and feel" similar to the C++ language, but it is easier to use than the C++ language and adopts a completely "object-oriented" approach to programming.

[0044] Structured Query Language (SQL) is a special-purpose programming language and a database query and programming language used to access data and query, update, and manage relational database systems.

[0045] Python provides efficient high-level data structures and simple and effective object-oriented programming. Python syntax and dynamic typing, as well as the nature of an interpreted language, make it a programming language for writing scripts and rapidly developing applications on most platforms.

[0046] Doris: Apache Doris is a high-performance, real-time analytical database based on the MPP architecture. It is well-known for its extremely fast and easy-to-use features. It can return query results under massive data in sub-second response time. It can not only support high-concurrency point query scenarios, but also support high-throughput complex analysis scenarios. Based on this, Apache Doris can better meet the use scenarios such as report analysis, ad hoc query, unified data warehouse construction, data lake federated query acceleration, etc. Users can build user behavior analysis, AB experimental platform, log retrieval analysis, user portrait analysis, order analysis and other applications on it.

[0047] GraalVM is a high-performance JDK designed to accelerate the execution of applications written in Java and other JVM languages, while also providing a runtime environment for other popular languages ​​such as JavaScript, Python, etc. GraalVM provides two ways to run Java applications, using the Graal Just-in-Time Compiler (JIT) on the HotSpot JVM or as a native executable compiled ahead of time (AOT). GraalVM's multi-language capabilities make it possible to mix multiple programming languages ​​in a single application while eliminating the cost of foreign language calls.

[0048] Arthas is an online monitoring and diagnosis product that can view the status information of application load, memory, GC, and threads in real time from a global perspective. It can also diagnose business problems without modifying the application code, including viewing the input and output parameters and exceptions of method calls, monitoring method execution time, class loading information, etc., greatly improving the efficiency of online problem troubleshooting.

[0049] Spring Cloud provides developers with tools to quickly build some common patterns in distributed systems (such as configuration management, service discovery, circuit breakers, intelligent routing, micro-agents, control buses, one-time tokens, global locks, leadership elections, distributed sessions, cluster states). The coordination of distributed systems uses boilerplate patterns, and using Spring Cloud developers can quickly build services and applications that implement these patterns. They will run well in any distributed environment, including the developer's own laptop, data center, and hosting platforms such as Cloud Foundry.

[0050] Quartz is a feature-rich open source job scheduling library that can be integrated into almost any Java application. From the smallest stand-alone application to the largest e-commerce system, Quartz can be used to create simple or complex time schedules to execute tens, hundreds, or even tens of thousands of jobs. Its tasks are defined as jobs of standard Java components that can perform almost any operation you might program them to. The Quartz scheduler includes many enterprise-level features, such as support for JTA transactions and clustering.

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] The data indicator management system provided by the present invention accesses the third-party data source through "database connection, third-party interface API" and other methods. Based on the process engine, data extraction and processing are executed in series, and then inquiries are provided to the outside in the form of "indicators", which simplifies the process that originally required complex modeling into the process of creating indicators. Specifically, Figure 2 As shown, the data indicator management system provided by the present invention includes: an indicator management platform, an indicator driving engine application, data processing related components and an indicator integration platform.

[0053] The indicator management platform is used to create indicators and store the created indicators in the platform database to implement maintenance and configuration of the platform database; the indicator-driven engine application is used to monitor the indicators stored in the platform database and trigger the indicator data processing process according to the maintenance configuration of the indicator management platform in the platform database; the data processing related components implanted with the indicator data processing application are used to initiate data requests to the target data source, and process the acquired indicator data according to the configuration to obtain indicator information; the indicator integration platform is used to integrate the indicator information according to the set conditions to generate report information.

[0054] Based on the above description, in the specific application process of the present invention, first, the indicator management platform uses the indicator management page to create the indicator, and the information edited and adopted in the creation process includes relevant information such as data source, trigger, data processing flow, etc. The indicator management platform can save the information to the platform database, and can monitor the indicator data in the platform database according to the indicator driving engine application (indicator driving engine), and when the trigger condition is reached during the detection process, the indicator processing process is automatically called, and local or third-party calls can also be made through the API. Among them, the indicator driving engine application mainly uses the automatic task component xxl-job to trigger the indicator data processing process based on the configuration maintained by the indicator management platform in the platform database. In the process of triggering the indicator production process, the dependent execution mode can be selected, that is, it can be executed according to the node sorting order, or non-dependent execution can be selected. This setting method is mainly to realize that each node is independently triggered and has different triggering conditions and time intervals, so as to adapt to different business needs.

[0055] The process of using the automatic task component xxl-job to trigger the indicator data processing flow is: merge the data of the node table with the xxl-job-info table in xxl-job. Each record of xxl-job-info represents an automatic task. When the automatic task is triggered, the relevant handler method will be executed. The handler integrates the node class and the relevant methods of the node, such as: judging the conditions for starting to obtain the result set, executing the result set acquisition, node status processing, data permission processing, etc., and judging and triggering according to the pre-set logic.

[0056] The configuration information managed by the management platform includes node-related information, such as data source ID and type, input parameters, next node ID, event execution ID, etc. related to data set acquisition, as well as attributes related to task scheduling, scheduling start time, scheduling frequency, etc. Nodes are combined with scheduling-related classes to achieve the effect of using the management platform to manage xxl-job tasks, and then using xxl-job to control node triggering and operation.

[0057] Furthermore, after the call starts, the multi-source data collection application implanted in the data processing related components initiates a data request to the target data source, so that the low-code platform application can perform fixed processing on the data according to different configurations. The low-code platform application mainly converts data through a preset data processing method, and uses the Sink operator to output the processed data to Redis.

[0058] In actual application, the function of multi-source data collection application is to obtain source data from the outside according to the configured data source information. The main integrated functions include reading database, API, message queue and file, etc. Database is obtained using JDBC, and other functions such as API are also processed using their corresponding components. In the process of obtaining source data, other indicator data can also be called as variables through the configuration information saved in the indicator management platform to realize the demand for dynamic source data. The implementation method of this process is as follows:

[0059] Relevant parameter data is obtained from the node in the form of a Key-Value structure, where the Key is used as a placeholder in the collection application in various configurations of data collection, such as presetting a date as a placeholder in the SQL statement. In the process of executing the collection, the indicator is first called using the API to obtain the result set of the parameters, and the aspect is used at each stage of data collection, such as SQL statements during SQL execution and file names in file processing to replace the placeholders, so as to achieve the effect of dynamic data source collection based on external dynamic data.

[0060] In actual application, low-code platform applications may have some fixed processing logic in the data processing process, such as row-to-column, column-to-row, splitting a single column into multiple rows, and other preset data processing logic for low-code processing. Low-code platforms mainly use flink, and use these preset logics as operators. After data processing, Sink is used to store the processed data in Redis for further processing.

[0061] At the same time, you can also use multilingual data processing applications (multilingual data platforms) to write data processing scripts in different programming languages ​​according to the user's programming habits, and also output the results to Redis. The output result is in the form of StringKey-JSONValue, where the Key is the unique identifier of the node or indicator, and the JSON format is fixed in the form of Array-Object. The Key of the Object is used as the column field of the indicator set, and the Value is the value of each row of the indicator set. Based on this, there are two types of RedisKey. The first is the node identifier, which is used to store the intermediate data of the result set of each node during the indicator execution process, which can be processed by the data processing application. The second is the indicator identifier, which is used to store the final result of the indicator result set and should be called for data modeling.

[0062] The subsequent processing logic is that when the indicator result set mark appears, the data modeling application will be called to model according to the result set and store the data.

[0063] In order to implement some complex data processing logic, multi-language data processing applications can process data indices by writing code during the data processing process. In order to adapt to the usage habits of different users, multi-language data processing applications use graalvm as JVM. Graalvm can support multiple programming languages ​​running in JVM, including JavaScript, Python, etc. If the customer has a high level of support for Python third-party libraries, there is also an alternative solution to use Python interpreter to execute script processing in multi-language data processing applications. The execution results will be stored in Redis, and the results saved in Redis can also be used when writing scripts.

[0064] Among them, indexation here means that users convert scattered data sets into data sets that conform to the business perspective based on business attributes. The specific plan is formulated by users from the technical perspective. The platform is responsible for providing support to users and integrating the indicators achieved by users from the technical perspective for use by users from the business perspective.

[0065] The process of processing the result set here is determined by the code entered by the user. The variable can be passed through the method parameter, or directly called in the form of redis in the respective code. The purpose of using graalvm is to target the Python language, and its parameters can also be saved to the JVM and directly used as parameters of its calling method. For example, if you want to use Python code to process the result set of the previous node, you can directly use the node ID of the previous node in the code to obtain the result set I need to process.

[0066] Furthermore, after the data is circulated, it is applied to the data modeling application, and the algorithm is used to analyze the existing indicator data, and a structured relational table is obtained, and the data is entered into the data warehouse in units of indicators. Among them, the main purpose of the data modeling application is to achieve the structuring of indicators. This process first uses the algorithm to process the processed indicator result set, and finally obtains a DDL data. When it is executed for the first time, the application will build a table in the data warehouse based on it. The purpose is to determine the dimensions of the data in the data warehouse, such as time, batch, etc., according to the configuration of the indicator management platform. This processing method makes it easier to trace back indicators.

[0067] The algorithm used when processing the indicator result set is mainly to find the greatest common divisor of the columns in the result set. Because the table structure of relational data requires all columns, the indicator results of each dimension are not necessarily complete, so when the result set is first obtained, all ObjectKeys of all columns in the result set need to be filtered out and processed as a union. After that, some ObjectValues ​​are obtained, and their data types and approximate sizes are determined by regular expression matching. After obtaining the information of these two, the dimension columns saved in the result set are obtained, and then the data can be structured.

[0068] The above table creation process is divided into two parts. The first part is that if the indicator table does not exist in the warehouse, according to the above process, after obtaining the field name and field type, the DDL statement can be generated. Then it is executed in the data warehouse. The second part is that if the table exists in the warehouse, it is judged according to the meta information of the table and the result set, and the part of the result set that has excess meta is obtained. The corresponding field name and type information is obtained to generate the DDL statement for table modification.

[0069] The indicator integration platform integrates the data in the data warehouse according to the indicators, data columns, dimensions and other related information required by users, and generates corresponding charts for business customers to use. Specifically, after the indicators are persisted, the indicator information can be integrated according to the indicators, fields, dimensions and other conditions in the indicator integration platform to generate report information, and this report information can be customized according to customer needs.

[0070] Furthermore, in addition to using structured indicator data, the indicator integration platform can also use Redis data to obtain some more real-time indicator results.

[0071] The purpose of the data indicator management system is to uniformly manage and maintain business-related data indicators in order to facilitate business monitoring, analysis and decision-making. The current data indicator management system solution includes indicator management, indicator-driven engine, multi-language indicator calculation, data modeling application and indicator integration platform, etc., which extracts, converts and stores data to unify data from different source systems to the target system, clean, process and model data, and provide support for subsequent data analysis and decision-making.

[0072] The following is a detailed introduction to the specific contents of the data indicator management system provided above in modules:

[0073] A. Indicator management platform: It realizes the management function of the entire indicatorization process and uses a relational database to provide persistence guarantee for the entire function. For example, the function of adding, deleting, modifying and checking indicator data can drive the entire data processing system through the indicator configuration data stored in the database. At the same time, the indicator management platform can also realize the indicatorization function of all data processes. By providing an API interface, the operating results of other indicators can be used in data processing related components or the execution process of other indicators can be called. The data of the entire system is fully indicatorized, allowing the system to create some more complex indicators. This application mainly provides support for each application in the flowchart to use other indicators. For example, in the process of collecting source data, you want to call the results of other indicators as parameters.

[0074] like Figure 3 As shown in the figure, other indicators can participate in each step of the indicator processing process. In view of this indicator data processing process, the indicator management platform designs a data structure based on a linked list to store the configuration and result data of each node in the process. In addition, the indicator management platform transmits and updates data in the data processing component based on logical processing functions such as logical judgment and data distribution, and finally saves it in the data warehouse.

[0075] At the same time, the indicator management platform also implements some basic management functions, such as version management and intermediate data recording of indicators based on the platform database and data warehouse, so as to facilitate users to troubleshoot possible problems in the indicator generation process.

[0076] B. Indicator-driven engine application: According to the configuration maintained by the indicator management platform in the platform database, the automatic task component xxl-job is used to trigger the indicator data processing process. During the triggering of the indicator production process, you can choose dependent execution mode (that is, execution according to the node sorting order) or non-dependent execution (that is, each node is triggered separately and has different triggering conditions and time intervals) to adapt to different business needs.

[0077] C. Multi-source data collection application: Its function is to obtain source data from the outside according to the configured data source information. The main integrated functions include reading databases, APIs, message queues, and files. Among them, the database is obtained using JDBC, and other functions such as APIs are also processed using their corresponding components. In the process of obtaining source data, other indicator data can also be called as variables through the configuration information saved in the indicator management platform to meet the needs of dynamic source data.

[0078] D. Low-code platform application: In the process of data processing, there may be some fixed processing logic, such as row-to-column, column-to-row, splitting a single column into multiple rows, and other preset data processing logic for low-code processing. The platform mainly uses flink, and uses these preset logics as operators. Finally, after processing, the results are stored in Redis using Sink for further processing.

[0079] E. Multi-language data processing applications: In order to implement some complex data processing logic, the data can be indexed by writing code during the data processing process. In order to adapt to the usage habits of different users, multi-language data processing applications use graalvm as JVM, which can support multiple programming languages ​​running in JVM, including JavaScript, Python, etc. There are two main reasons for using this solution. On the one hand, its support for JavaScript is better than the built-in engine of JVM. On the other hand, the intermediate results are integrated into the JVM during the calculation process, which is more conducive to the use of temporary variables saved in the JVM when writing scripts. If the customer has a high support for Python third-party libraries, there should also be an alternative solution to use the Python interpreter to execute script processing. The execution results will be stored in Redis, and the results saved in Redis can also be used when writing scripts.

[0080] F. Data modeling application: The main purpose of this application is to achieve the structuring of indicators. When implemented, the processed indicator result set is first processed using an algorithm to finally obtain a DDL data. Then, when executed, the application will build a table in the data warehouse based on it. Its purpose is to determine the dimensions of the data in the data warehouse, such as time, batch, etc., based on the configuration of the indicator management platform. This processing method makes it easier to trace back indicators.

[0081] Furthermore, there are two ways to solidify indicators. The first is to use memory cache such as Redis, which can cache indicator result sets to relieve the pressure on the data source end. Users can save indicator data with high real-time requirements, and it can also be used as a message queue for data communication within the indicator process. The second is to use the MPP database Doris as a persistent storage solution and retain the history of indicators by dimension.

[0082] G. Indicator integration platform: After the indicators are persisted, the indicator information can be integrated according to indicators, fields, dimensions and other conditions in the platform to generate report information, and the report information can be customized according to customer needs.

[0083] Based on the above description, compared with the prior art, the present invention also has the following advantages:

[0084] 1) In terms of user experience, the needs of customers from different perspectives can be met. The present invention provides a relatively complete indicator management system and an indicator analysis system, which facilitates the user's IT department and business department to use indicators from different perspectives.

[0085] 2) The system of the present invention can be fully decoupled, and the functional modules are highly extensible, which can meet various needs of different users' on-site conditions. For example, multi-source data collection applications and multi-language data processing applications integrate a variety of common processing methods, and maintain the plug-and-play feature in terms of code, so that the cost of modification for special usage scenarios is relatively low.

[0086] 3) The data storage structure of the present invention is relatively friendly. Different from the large wide table mode of the traditional data warehouse, the present invention adopts a multi-table storage structure divided according to indicators, which makes the data structure of each indicator relatively clear and also facilitates the use of different dimensions to store the horizontal data of the table.

[0087] 4) The present invention can realize the indexing of data flow. The present invention can index all contents related to data, for example, a certain date in an SQL statement can be passed in using a certain index as a parameter.

[0088] 5) The system in the present invention has high availability. A distributed system solution is used to decompose each link into different services, and load balancing is used to ensure the availability of each node.

[0089] 6) The present invention can construct a process by dragging the front-end page, and uses some default algorithms to lower the threshold for manually constructing the process.

[0090] 7) The present invention has multiple alternatives for accepting external data and outputting process data, which can ensure the versatility and stability of the process.

[0091] 8) The present invention uses a method of retaining intermediate data of process execution in memory, so as to facilitate rapid location of problematic nodes when problems occur in the process.

[0092] 9) When the source data changes, the present invention does not need to operate the view. The entire process is automatically recognized by the program and the entire view is automatically and incrementally reloaded through the view layout engine and the view algorithm engine.

[0093] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0094] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A data indicator management system, characterized in that: include: The indicator management platform is used to create indicators and store the created indicators in the platform database to implement the maintenance and configuration of the platform database; An indicator-driven engine application is used to monitor the indicators stored in the platform database and trigger the indicator data processing flow according to the maintenance configuration of the indicator management platform in the platform database; Data processing related components are embedded with indicator data processing applications, which are used to initiate data requests to the target data source and process the obtained indicator data according to the configuration to obtain indicator information; An indicator integration platform, used to integrate the indicator information according to set conditions to generate report information; When the indicator-driven engine application monitors the platform database data, when the trigger condition is reached, the indicator processing process is automatically called, and local or third-party calls are made through the API; The indicator-driven engine application uses the automatic task component xxl-job to trigger the indicator processing flow, and selects an execution mode during the indicator processing flow; the execution mode includes a dependent execution mode and an independent execution mode; The indicator data processing application includes: Multi-source data collection application, used to obtain source data from the outside according to the configured data source information; in the process of obtaining source data, other indicator data are called as variables through the configuration information saved in the indicator management platform to realize the dynamic demand of source data; Low-code platform applications have fixed processing logic in the data processing process, which is used to process data, and after data processing, Sink is used to store the processed data in Redis for further processing; the fixed processing logic includes: row-to-column, column-to-row, and splitting a single column into multiple rows; Multilingual data processing applications, used to index data by writing code during the data processing process; Data modeling applications to structure indicators; Multi-language data processing applications use graalvm as JVM to support multiple programming languages ​​running in JVM; in multi-language data processing applications, use the Python interpreter as an alternative to execute script processing and store the execution results in Redis; use the execution results saved in Redis when writing scripts.

2. The data indicator management system according to claim 1, characterized in that: The indicator management platform uses an indicator management page to create indicators; the editing information performed during the indicator creation process includes data source information, trigger information, and data processing flow information.

3. The data indicator management system according to claim 1, characterized in that: The functions of the multi-source data collection application integration include: reading databases, APIs, message queues and files; wherein the database is acquired using JDBC, and the API is processed using components corresponding to the API.

4. The data indicator management system according to claim 1, characterized in that: The structured implementation process of indicators is as follows: the processed indicator result set is processed using an algorithm to obtain a DDL data; when executed for the first time, the data modeling application is used to create a table in the data warehouse to achieve the purpose of determining the data dimension in the data warehouse based on the configuration of the indicator management platform.

5. The data indicator management system according to claim 1, characterized in that: The indicator integration platform customizes and displays the generated report information according to customer needs.

6. The data indicator management system according to claim 1, characterized in that: The indicator integration platform is also used to obtain indicator results using Redis data.

Images