Method for dynamically configuring indexes and generating data
Through dynamic configuration and generation of indicator data, the problem of re-encoding of indicator data changes in traditional software development is solved, and rapid response and efficient indicator development is achieved, and business efficiency and flexibility are improved.
Patent Information
- Application Number
- CN202411163406.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-01
AI Technical Summary
In traditional software development, the statistical analysis of indicator data needs to clarify the data source, structural fields and calculation methods, which requires re-encoding when the indicator data changes, affecting business efficiency, especially when the external interface data is adjusted for a longer time.
Provide a method of dynamic configuration of indicators and generating data. By accessing different data sources, defining indicator update cycles, embedding dynamic scripts, and batch calling indicator calculations in thread pools are used to poll, providing convenient indicator query interfaces, supporting Dynamic configuration and calculation of basic and composite indicators.
It greatly shortens the time for indicator development and change, from days to hours or even minutes, improves business efficiency and flexibility, reduces trial and error costs, and does not affect system stability and accessibility.
Smart Images

Figure CN118885486A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a method for dynamically configuring indicators and generating data. Background Art
[0002] Generally speaking, indicator data is basically based on various detailed data of the system. The indicator data is obtained through statistical analysis and output to the front-end interface for display. The statistical analysis here requires clarifying the data source, structure field and calculation method required for the indicator in traditional software development, and encoding them into the program. In this way, once the indicator data source changes or the calculation method changes, it is necessary to recode and deploy the program. It may take 0.5-1 day to adjust and develop a single indicator, and once it involves the adjustment of external interface data, it may take longer, affecting business development.
[0003] Therefore, a method to dynamically configure indicators and generate data has become an urgent problem to be solved. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a method for dynamically configuring indicators and generating data. By connecting to different data sources, defining the indicator update cycle and providing an update interface, dynamic scripts are embedded for easy expansion, the indicator calculation results are stored, and a convenient indicator query interface is provided to the outside world.
[0005] To solve the above technical problems, the present invention provides a technical solution: a method for dynamically configuring indicators and generating data, comprising the following steps:
[0006] S1. Data source management
[0007] Data sources include databases and external interfaces;
[0008] Database management is used to manage storage connection information, and external interface management is used to manage accessible external interface information;
[0009] S2. Indicator definition configuration
[0010] The indicators are divided into basic indicators and composite indicators according to their value sources; basic information of basic indicators and composite indicators are configured respectively;
[0011] S3, indicator calculation engine
[0012] The indicator calculation engine periodically discovers indicators to be updated, obtains data based on indicator call data sources, calls scripts to calculate indicator data, and saves indicator data;
[0013] The indicator calculation engine uses polling to obtain indicators to be updated from the indicator library and calls them in batches in the thread pool;
[0014] After the indicator calculation engine calculates the data of each indicator and obtains the result, it changes the next update time of the indicator according to the update cycle of the indicator;
[0015] S4, indicator data interface
[0016] The indicator data interface provides indicator data to the outside world. The provided indicator data is simply aggregated according to the actual configuration;
[0017] The indicator data interface needs to be configured and provide a unique call code for external calls. The configuration information includes: call code, aggregated indicator list;
[0018] According to the configuration, when the indicator data interface is called externally, it queries all indicator data in the aggregated indicator list and returns it.
[0019] Furthermore, the connection information in step S1 includes a connection address, a user name, a password, a library name, and other necessary parameters;
[0020] The external interface information in step S1 includes the address of the interface, the request method, the request parameters, whether to paginate, the pagination parameter definition, the front script, and the authentication information.
[0021] Furthermore, in step S1, the database management also provides a dynamic connection and reuse mechanism for each database, that is, when the database needs to be used, if it is not connected, the connection information is obtained to connect, and if it is already connected, the connection is reused.
[0022] Furthermore, when defining the basic indicators in step S2, the indicator name, data source, single-page statistical script, data integration script, and update cycle are configured;
[0023] When defining the composite indicator, the indicator name, dependent indicator list, calculation script, and update cycle are configured.
[0024] Furthermore, when defining the basic indicators in step S2, if the data source is a database source, the database table, data acquisition logic and paging parameters are configured; the statistical script obtains the data returned by the data source, passes it into the script, and executes it to obtain the final single-page statistical data; if there is paging, after all paging statistical data are calculated, the data integration script is executed to integrate the statistical single-page data to obtain the final data; after the indicator data is obtained, it should be saved for direct reading next time.
[0025] Furthermore, in step S3, the basic indicators are called in different ways according to different data sources.
[0026] Furthermore, when the data source of the basic indicator is a database source:
[0027] If the basic indicator is configured with data acquisition logic, a SQL script is generated based on the data acquisition logic; if it is a SQL script, it is directly acquired;
[0028] Initialize and compile the single-page statistics script for subsequent repeated calls;
[0029] Get the corresponding database information and obtain the connection. If successful, assemble and execute the SQL script according to the paging situation to obtain data;
[0030] The acquired data is passed into the compiled single-page statistical script to obtain the calculation results and put them into the temporary storage list;
[0031] Change the parameters of the next page and assemble and execute a new SQL script to obtain data. If there is data, repeat the previous calculation action. If there is no data, jump out of the loop of the next page.
[0032] The data in the temporary storage list is passed into the data integration script to obtain the calculation results, which are stored in the indicator result table and the indicator result history table.
[0033] Furthermore, when the data source of the basic indicator is an external interface source:
[0034] If the authentication method configured by the basic indicator is basic authentication or key authentication, it is directly used as a parameter; if the authentication method configured by the basic indicator is interface authentication, the authentication interface is called, and the returned data is used as a parameter, which is passed into the interface front-end script, and the paging data and the complete request object are passed in for the script to modify;
[0035] Initialize and compile the single-page statistics script for subsequent repeated calls;
[0036] Use the request object modified by the script to initiate a data request and obtain the data;
[0037] The acquired data is passed into the compiled single-page statistical script to obtain the calculation results and put them into the temporary storage list;
[0038] Change the parameters of the next page and re-transmit them to the interface front-end script as paging data. Call the interface to obtain new data. If there is data, repeat the previous calculation action. If there is no data, jump out of the loop of the next page.
[0039] The data in the temporary storage list is passed into the data integration script to obtain the calculation results, which are stored in the indicator result table and the indicator result history table.
[0040] Furthermore, in step S3, the composite index calculation calling method is as follows:
[0041] Get all the basic indicators and corresponding results that the composite indicator depends on, match the basic indicators with the corresponding results one by one, pass them into the calculation script for calculation, and get the calculation results, and store them in the indicator result table and the indicator result history table.
[0042] Furthermore, indicator definitions are allowed to be updated and deleted;
[0043] When updating the indicator definition, update the configuration information corresponding to the indicator and store the original configuration in the historical configuration table to facilitate quick query or recovery of the historical configuration;
[0044] When deleting an indicator definition, only logical deletion is performed. This logical deletion is only effective for scheduled calculations. If the query interface passes in specified parameters, the indicator data can still be queried.
[0045] The advantages of the present invention over the prior art are: after implementing the technical solution of the present invention, the work related to data statistics and indicator display no longer needs to be developed independently, and can be completed by performing corresponding configuration in the system implemented by the method of the present invention. The development and modification of indicators can be shortened from several days to several hours or even minutes, which greatly increases the efficiency and flexibility of business docking. And based on the ability of rapid configuration, the cost of trial and error can be reduced, especially for indicator parameters that are not yet fully clear for the business, which can be changed frequently and quickly without affecting the stability and accessibility of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 The present invention is a flowchart of a method for dynamically configuring indicators and generating data.
[0047] Figure 2 It is an operational flow chart that uses the database as the data source to dynamically configure basic indicators and generate data.
[0048] Figure 3 It is an operational flow chart that uses external interfaces as data sources to dynamically configure basic indicators and generate data.
[0049] Figure 4 It is an operational flow chart that uses the database as the data source to dynamically configure basic indicators and generate data when using dynamic scripts to process complex scenarios.
[0050] Figure 5 It is an operational flow chart that uses external interfaces as data sources to dynamically configure basic indicators and generate data when using dynamic scripts to process complex scenarios.
[0051] Figure 6 It is an operational flow chart for dynamically configuring composite indicators and generating data. DETAILED DESCRIPTION
[0052] Various exemplary embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention.
[0053] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.
[0054] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.
[0055] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.
[0056] Combined with Figure 1-6 , the present invention is introduced in detail.
[0057] A method for dynamically configuring indicators and generating data, including data source management, indicator definition configuration, indicator calculation engine and indicator data interface;
[0058] The specific steps are as follows:
[0059] 1. Data source management
[0060] Data sources fall into two categories: databases and interfaces.
[0061] (1) Database management
[0062] Database management mainly manages other database information that can be connected, including the necessary information for database connection: connection address, user name, password, database name and other necessary parameters. For security reasons, the password can be encrypted and stored using a symmetric encryption algorithm.
[0063] In addition to storing database connection information, the database management of this method also provides a dynamic connection and reuse mechanism for each database, that is, when the database needs to be used, if it is not connected, the connection information is obtained to connect, and if it is already connected, the connection is reused, so that operations can be performed faster.
[0064] (2) Interface Management
[0065] Interface management mainly manages the accessible external interface information, including the interface address, request method, request parameters, whether to paginate, pagination parameter definition, pre-script, authentication information, etc. Among them, the authentication information can be a fixed username and password or other fixed authentication information, or specify other interfaces as authentication interfaces, and use the data returned by the authentication interface as authentication information.
[0066] On the authentication interface, commonly used interface authentication methods are integrated, such as basic authentication (basic), token authentication (obtaining tokens through the authentication interface), key authentication (fixed key), and custom authentication methods.
[0067] At the same time, commonly used encryption algorithms are provided, and various encryption algorithms can be supplemented at any time as needed to facilitate data signing or encryption and decryption calculations in some interfaces.
[0068] In the front script of the interface, dynamic scripting languages such as javascript and lua are embedded, and encryption algorithms can be used to easily realize dynamic access to various interfaces.
[0069] 2. Indicator definition configuration
[0070] Indicators are divided into basic indicators and composite indicators according to their value sources, and this method does not involve indicator classification and grading. Almost all indicators can be obtained by combining basic indicators and composite indicators.
[0071] (1) Basic indicators
[0072] Basic indicators refer to indicators that can be obtained directly through statistical analysis of data sources, such as directly obtained through database queries or data returned by interfaces, or obtained by aggregating data returned by interfaces.
[0073] When defining basic indicators, you need to configure information such as indicator name, data source, single-page statistical script, data integration script, update cycle (real-time, scheduled), etc.
[0074] The data source should be selected from the data source configured in the data source management. If it is a database source, configure the database table, data acquisition logic (which can be a SQL script or a definition of data acquisition, which can eventually be converted into a SQL script for execution) and paging parameters; the statistical script obtains the data returned by the data source, passes it into the script, and executes it to obtain the final single-page statistical data; if there is paging, after all paging statistical data are calculated, execute the data integration script to integrate the statistical single-page data to obtain the final data.
[0075] After the indicator data is obtained, it should be saved so that it can be directly read next time.
[0076] (2) Composite indicators
[0077] Composite indicators refer to indicators that do not obtain data directly from data sources, but are calculated by integrating basic indicators or other composite indicators.
[0078] When defining a composite indicator, you need to configure the indicator name, dependent indicator list, calculation script, update cycle and other information.
[0079] The calculation script calculates the data of this composite indicator by passing in the list of dependent indicators and the corresponding indicator data, and saves it for direct reading next time.
[0080] 3. Index calculation engine
[0081] The indicator calculation engine mainly refers to the engine that drives the calculation of indicator data. The engine is responsible for regularly discovering indicators to be updated, obtaining data based on indicator data sources, calling scripts to calculate indicator data, and saving indicator data.
[0082] The engine uses polling to obtain the indicators to be updated from the indicator library and makes batch calls in the thread pool.
[0083] After the engine calculates the data of each indicator and obtains the result, it changes the next update time of the indicator according to the update cycle of the indicator.
[0084] (1) Basic indicator calculation call
[0085] Basic indicators are called in different ways according to different data sources.
[0086] ①Database source
[0087] If the basic indicator is configured with data acquisition logic, a SQL script is generated based on the data acquisition logic; if it is a SQL script, it is directly acquired.
[0088] Initialize and compile the single-page statistics script for subsequent repeated calls.
[0089] Get the corresponding database information and obtain the connection. If successful, assemble and execute the SQL script according to the paging situation to obtain data.
[0090] The acquired data is passed into the compiled single-page statistics script to obtain the calculation results and put them into the temporary storage list.
[0091] Change the parameters of the next page and assemble and execute a new sql (next page) script to obtain data. If there is data, repeat the previous calculation action. If there is no data, jump out of the loop of the next page.
[0092] The data in the temporary storage list is passed into the data integration script to obtain the calculation results, which are stored in the indicator result table and the indicator result history table.
[0093] ②Indicator data interface
[0094] If the authentication method of the basic indicator configuration is basic authentication or key authentication, it is directly used as a parameter; if the authentication method of the basic indicator configuration is interface authentication, the authentication interface is called, and the returned data is used as a parameter, which is passed into the interface front-end script, and the paging data and the complete request object are passed in for the script to modify.
[0095] Initialize and compile the single-page statistics script for subsequent repeated calls.
[0096] Use the request object modified by the script to initiate a data request and obtain the data.
[0097] The acquired data is passed into the compiled single-page statistics script to obtain the calculation results and put them into the temporary storage list.
[0098] Change the parameters of the next page and re-transmit them to the interface front-end script as paging data. Call the interface to obtain new data. If there is data, repeat the previous calculation action. If there is no data, jump out of the loop of the next page.
[0099] The data in the temporary storage list is passed into the data integration script to obtain the calculation results, which are stored in the indicator result table and the indicator result history table.
[0100] (2) Composite index calculation call
[0101] Get all the indicators and corresponding results that the composite indicator depends on, match the indicators and results one by one, pass them into the calculation script for calculation, and get the calculation results, which are stored in the indicator result table and the indicator result history table.
[0102] 4. Index data interface
[0103] The indicator data interface is responsible for providing indicator data to the outside world. The provided indicator data can be simply aggregated according to the actual configuration.
[0104] The data interface also needs to be configured and provide a unique calling code for external calls. The configuration information may include: calling code (customizable or generated by the system according to actual needs), aggregated indicator list and other information.
[0105] According to the configuration, when the data interface is called externally, it can query all indicator data in the aggregated indicator list and return it.
[0106] The specific implementation process of a method for dynamically configuring indicators and generating data of the present invention is as follows:
[0107] 1. General Data Preparation
[0108] Database Management
[0109] General scenario: Create a new table table_db in the database db_one and establish the data structure
[0110] {
[0111] "field_id":"int",
[0112] "field_name":"varchar(100)",
[0113] "field_ip_port":"varchar(255)",
[0114] "field_db_type":"varchar(50)",
[0115] "field_user_name":"varchar(100)",
[0116] "field_pass_word":"varchar(100)",
[0117] "field_create_time":"datetime",
[0118] "field_update_time":"datetime",
[0119] "field_del_flag":"varchar(2)"
[0120] }
[0121] CREATE TABLE table_db(
[0122] field_id INT PRIMARY KEY AUTO_INCREMENT,
[0123] field_name VARCHAR(100)NOT NULL,
[0124] field_ip_port VARCHAR(255)NOT NULL,
[0125] field_db_type VARCHAR(50)NOT NULL,
[0126] field_user_name VARCHAR(100)NOT NULL,
[0127] field_pass_word VARCHAR(100)NOT NULL,
[0128] field_create_time DATETIME DEFAULT CURRENT_TIMESTAMP,
[0129] field_update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATECURRENT_TIMESTAMP,
[0130] field_del_flag VARCHAR(2)DEFAULT'0' );
[0132] External interface management
[0133] General scenario: Create a new table table_api in the database db_one and establish the data structure
[0134] For the sake of convenience, the system agrees to use the data structure as JSON:
[0135] {
[0136] "api_id":"int",
[0137] "api_name":"varchar(100)",
[0138] "api_url":"varchar(255)",
[0139] "api_method":"varchar(10)",
[0140] "api_header":"varchar(500)",
[0141] "api_body":"text",
[0142] "api_token_status":"varchar(2)",
[0143] "api_pre_call_script:"varchar(500)",
[0144] "api_token_url":"varchar(500)",
[0145] "api_paginated":"varchar(2)",
[0146] "api_get_next_page": "varchar(500)",
[0147] "create_time": "datetime",
[0148] "update_time": "datetime",
[0149] "del_flag": "varchar(2)"
[0150] }
[0151] Table structure definition:
[0152] CREATE TABLE table_api(
[0153] api_id INT PRIMARY KEY AUTO_INCREMENT,
[0154] api_name VARCHAR(100) NOT NULL,
[0155] api_url VARCHAR(255) NOT NULL,
[0156] api_method VARCHAR(10) NOT NULL,
[0157] api_header VARCHAR(500),
[0158] api_body TEXT,
[0159] "api_token_status": "varchar(2)",
[0160] "api_pre_call_script": "varchar(500)",
[0161] "api_token_url": "varchar(500)",
[0162] "api_paginated": "varchar(2)",
[0163] "api_get_next_page": "varchar(500)",
[0164] create_time DATETIME DEFAULT CURRENT_TIMESTAMP,
[0165] update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,del_flag VARCHAR(2)DEFAULT'0' );
[0167] Indicator Management
[0168] General scenario: Create a new indicator class table in the database db_one and generate the data structure
[0169] 1. Basic indicator table base_metric
[0170] CREATE TABLE base_metric(
[0171] base_metric_id INT PRIMARY KEY AUTO_INCREMENT,
[0172] base_metric_name VARCHAR(100)NOT NULL,
[0173] current_version VARCHAR(50)NOT NULL,
[0174] create_time DATETIME DEFAULT CURRENT_TIMESTAMP,
[0175] update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP);
[0176] 2. Basic indicator version table base_metric_version
[0177] CREATE TABLE base_metric_version(
[0178] base_metric_version_id INT PRIMARY KEY AUTO_INCREMENT,
[0179] base_metric_id INT NOT NULL,
[0180] base_metric_name VARCHAR(100) NOT NULL,
[0181] data_source VARCHAR(255) NOT NULL,
[0182] table_name VARCHAR(100) NOT NULL,
[0183] data_field VARCHAR(100) NOT NULL,
[0184] fetch_script TEXT,
[0185] statistic_field VARCHAR(100) NOT NULL,
[0186] statistic_method VARCHAR(100) NOT NULL,
[0187] update_frequency VARCHAR(50) NOT NULL,
[0188] create_time DATETIME DEFAULT CURRENT_TIMESTAMP,
[0189] update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, FOREIGN KEY(base_metric_id) REFERENCES base_metric(base_metric_id) );
[0191] 3. Composite index table composite_metric
[0192] CREATE TABLE composite_metric(
[0193] composite_metric_id INT PRIMARY KEY AUTO_INCREMENT,
[0194] composite_metric_name VARCHAR(100) NOT NULL,
[0195] current_version VARCHAR(50) NOT NULL,
[0196] create_time DATETIME DEFAULT CURRENT_TIMESTAMP,
[0197] update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP);
[0198] 4. Composite Metric Version Table composite_metric_version
[0199] CREATE TABLE composite_metric_version(
[0200] composite_metric_version_id INT PRIMARY KEY AUTO_INCREMENT,
[0201] composite_metric_id INT NOT NULL,
[0202] composite_metric_name VARCHAR(100) NOT NULL,
[0203] related_base_metric VARCHAR(255) NOT NULL,
[0204] related_composite_metric VARCHAR(255),
[0205] calculation_formula TEXT NOT NULL,
[0206] update_frequency VARCHAR(50) NOT NULL,
[0207] create_time DATETIME DEFAULT CURRENT_TIMESTAMP,
[0208] update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,FOREIGN KEY(composite_metric_id)REFERENCES composite_metric(composite_metric_id) );
[0210] {
[0211] "table":"composite_metric_version",
[0212] "composite_metric_id":1,
[0213] "composite_metric_name":"ExampleCompositeMetricVersion",
[0214] "related_base_metric":"1,2",
[0215] "related_composite_metric":"3",
[0216] "calculation_formula":"SUMmetric1,metric2) / metric3",--Error:'SUMmetric1,metric2) / metric3' is not a valid calculation formula format
[0217] "update_frequency":"WEEKLY"
[0218] }
[0219] 2. Practical Examples
[0220] Combined with Figure 2 , data source instance
[0221] 1. Data source definition: A set of list data is stored in table a of database A.
[0222] 2. Definition of basic indicator ta: The basic indicator ta is the result of adding and summarizing the ac fields of table a.
[0223] step:
[0224] 1. Select the indicator: Select the corresponding basic indicator ta. The system searches for the corresponding version information of ta from the base_metric_version table and obtains data_source and fetch_script.
[0225] 2. Query the corresponding database: Get data_source from the base_metric_version table. For example, data_source contains information for connecting to database A. The system uses the information in data_source to find the database connection information in the table_db table.
[0226] 3. Dynamically switch databases: Based on the database connection information obtained from the table_db table, the system dynamically switches the data source and connects to database A.
[0227] 4. Execute the script: Get the SQL query script fetch_script to be executed from the base_metric_version table. For example, fetch_script is SELECT SUM(ac) FROM a. Execute this SQL query script in the dynamically switched database connection.
[0228] 5. Get indicators: Get the result data of SQL query. For example, the query result is 1000.
[0229] 6. Store in the database: Store the obtained indicator data 1000 in the metric_result table.
[0230] 7. Return the specific indicator data name and value through the interface. End: The system returns the specific indicator data name ta and its value 1000 through the interface.
[0231] Specific example:
[0232] 1. Specify the indicator ta.
[0233] 2. The system finds the following information in the base_metric_version table:
[0234] o data_source: {"db_name":"databaseA","table":"a"}
[0235] o fetch_script:"SELECT SUM(ac)FROM a"
[0236] 3. The system searches the table_db table and finds the connection information: {"host":"db_host","user":"db_user","password":"db_pass"}.
[0237] 4. The system connects to database databaseA and executes the script SELECT SUM(ac) FROM a.
[0238] 5. The query result is 1000, and the system stores it in the metric_result table.
[0239] 6. The system returns the value of indicator ta as 1000.
[0240] Combined with Figure 3 , interface source instance
[0241] 1. Data source definition: The external interface returns a paginated data list.
[0242] 2. Definition of basic indicator tx: The basic indicator tx is the average value of all data fields x returned by the statistical interface.
[0243] step:
[0244] 1. Select the indicator: Select the corresponding basic indicator tx. The system searches the base_metric_version table for the version information corresponding to tx and obtains data_source and fetch_script.
[0245] 2. Query the corresponding external interface: Get data_source from the base_metric_version table. For example, data_source contains the URL and request method of the external interface. The system uses the information in data_source to find the interface information in the table_api table.
[0246] 3. Send request: Based on the interface information obtained from the table_api table, the system sends an HTTP request to obtain a paginated data list. For example, the data returned by the interface is a JSON array containing multiple objects, each of which has a field x.
[0247] 4. Execute the script: Get the query script fetch_script to be executed from the base_metric_version table. Parse the data returned by the interface and calculate the average value of all data fields x. For example, fetch_script defines how to parse JSON data and calculate the average value.
[0248] 5. Get indicators: Get the result data of script execution. For example, the calculated average value is 75.
[0249] 6. Store in the database: Store the obtained indicator data 75 in the metric_result table.
[0250] 7. Return the specific indicator data name and value through the interface. End: The system returns the specific indicator data name tx and its value 75 through the interface.
[0251] Specific example:
[0252] 1. Specify the indicator tx.
[0253] 2. The system finds the following information in the base_metric_version table:
[0254] o data_source: {"url":"http: / / api.example.com / data","method":"GET"}
[0255] o fetch_script: "JSON.parse(response).map(item=>item.x).reduce((a,b)=>a+b) / JSON.parse(response).length"
[0256] 3. The system searches the table_api table and finds the interface information: {"url":"http: / / api.example.com / data","method":"GET"}.
[0257] 4. The system sends an HTTP request and obtains the following data: [
[0259] {"x":60},
[0260] {"x":70},
[0261] {"x":80} ]
[0263] 5. The system executes the script, calculates the average value as 70, and stores it in the metric_result table.
[0264] 6. The system returns the value of indicator tx, which is 70.
[0265] 3. Complex scenarios processed by dynamic scripts
[0266] Definition: Basic indicators calculated by loading dynamic scripts. The scripts use dynamic script engines, and need to pass in parameters and get results.
[0267] Combined with Figure 4 , database data source
[0268] 1. Select the indicator: Select the corresponding basic indicator. The system searches for the corresponding version information from the base_metric_version table and obtains data_source and fetch_script.
[0269] 2. Query the corresponding data source: Get data_source from the base_metric_version table. For example, data_source contains database information. The system obtains the corresponding data based on data_source.
[0270] 3. Execute data query: Based on the data obtained by data_source, the system executes query operations to obtain the required raw data. For example, a list of data that needs weighted average is obtained.
[0271] 4. Loading dynamic script engine: The system loads the corresponding dynamic script engine according to the dynamic script defined by fetch_script.
[0272] 5. Pass in parameters and execute scripts: Pass the acquired raw data as parameters to the dynamic script engine. The script engine performs calculations based on the passed in parameters, for example, calculating weighted averages. The script needs to conform to the defined input and output parameter methods. Usually, the input parameter can be a JSON object or array, and the output parameter is the result value of the calculation.
[0273] Input parameters:
[0274] {
[0275] "data":[
[0276] {"value":10,"weight":0.2},
[0277] {"value":20,"weight":0.3},
[0278] {"value":30,"weight":0.5} ]
[0280] }
[0281] Parameters:
[0282] {
[0283] "result":22.0
[0284] }
[0285] 6. Get indicators: Get the execution result data from the dynamic script engine. For example, the calculated weighted average is 22.0.
[0286] 7. Determine whether to continue processing: The system can determine whether to further process the result data based on business needs. For example, if the calculation result meets certain conditions, it will trigger subsequent processing processes such as alarms, data storage, or other operations.
[0287] 8. Store in the database: Store the obtained indicator data 22.0 in the metric_result table.
[0288] 9. Return the specific indicator data name and value through the interface. End: The system returns the specific indicator data name and its value 22.0 through the interface.
[0289] Specific example:
[0290] 1. Specify the corresponding indicators.
[0291] 2. The system finds the following information in the base_metric_version table:
[0292] o data_source: {"db_name":"databaseB","table":"b"}
[0293] o fetch_script: "function calculateWeightedAverage(data){let sum=0; let totalWeight=0;
[0294] data.forEach(item=>{sum+=item.value*item.weight; totalWeight+=item.weight;}); return sum / totalWeight;}"
[0295] 3. The system searches the table_db table and finds the connection information: {"host":"db_host","user":"db_user","password":"db_pass"}.
[0296] 4. The system connects to database databaseB and executes a query to obtain the following data: [
[0298] {"value":10,"weight":0.2},
[0299] {"value":20,"weight":0.3},
[0300] {"value":30,"weight":0.5} ]
[0302] 5. The system loads the JavaScript engine and executes the script, calculating the weighted average as 22.0 and storing it in the metric_result table.
[0303] 6. The system returns the indicator value 22.0.
[0304] Combined with Figure 5 , interface data source
[0305] 1. Select the indicator: Select the corresponding basic indicator. The system searches for the corresponding version information from the base_metric_version table and obtains data_source and fetch_script.
[0306] 2. Query the corresponding external interface: Get data_source from the base_metric_version table. For example, data_source contains the URL and request method of the external interface. The system uses the information in data_source to find the interface information in the table_api table.
[0307] 3. Processing request parameters: The system processes HTTP requests according to requirements, for example, adding basic authentication or performing MD5 encryption.
[0308] Pre-request processing example:
[0309] function preprocessRequest(params){
[0310] / / Add Basic authentication
[0311] let username = 'user';
[0312] let password = 'pass';
[0313] let auth='Basic'+btoa(username+':'+password);
[0314] / / MD5 encryption
[0315] let md5Hash = md5(params);
[0316] return {
[0317] headers:{
[0318] 'Authorization':auth,
[0319] 'Content-MD5':md5Hash
[0320] },
[0321] body:params
[0322] };
[0323] }
[0324] 4. Send request: According to the processed request parameters, the system sends an HTTP request to obtain a paginated data list. For example, the data returned by the interface is a JSON array containing multiple objects, each of which has a field x.
[0325] 5. Execute the script: Get the query script fetch_script to be executed from the base_metric_version table. Parse the data returned by the interface and calculate the average value of all data fields x. For example, fetch_script defines how to parse JSON data and calculate the average value.
[0326] 6. Get indicators: Get the result data of script execution. For example, the calculated average value is 75.
[0327] 7. Store in the database: Store the obtained indicator data 75 in the metric_result table.
[0328] 8. Return the specific indicator data name and value through the interface. End: The system returns the specific indicator data name and value 75 through the interface.
[0329] Specific example:
[0330] 1. Specify the corresponding indicators.
[0331] 2. The system finds the following information in the base_metric_version table:
[0332] o data_source: {"url":"http: / / api.xxx.com / xxx","method":"POST"}
[0333] o fetch_script: "JSON.parse(response).map(item=>item.x).reduce((a,b)=>a+b) / JSON.parse(response).length"
[0334] 3. The system searches the table_api table and finds the interface information: {"url":"http: / / api.xxx.com / xxx","method":"POST"}.
[0335] 4. The system processes the request, such as adding Basic authentication and MD5 encryption, and generates processed request parameters.
[0336] 5. The system sends an HTTP request and obtains the following data: [
[0338] {"x":60},
[0339] {"x":70},
[0340] {"x":80} ]
[0342] 6. The system executes the script, calculates the average value as 70, and stores it in the metric_result table.
[0343] 7. The system returns the indicator value 70.
[0344] 4. Composite indicator example
[0345] Combined with Figure 6 , Example
[0346] 1. Data source definition: result data of multiple basic indicators.
[0347] 2. Definition of composite index: A composite index is the weighted sum of multiple basic indicators.
[0348] step:
[0349] 1. Select the metric: Select the corresponding composite metric. The system searches for the corresponding version information in the composite_metric_version table and obtains the components and formula.
[0350] 2. Get basic indicator data: The system gets the required basic indicators from components. For example, the basic indicators are multiple different indicators. The system queries the metric_result table to get the result data of these basic indicators.
[0351] 3. Read calculation formula: Read the calculation formula from the formula field. For example, the formula is ta*0.5+tb*0.3+tc*0.2.
[0352] 4. Perform calculation: Substitute the obtained basic indicator data into the calculation formula for calculation. For example, calculate the weighted sum.
[0353] 1. Get indicators: Get the calculated result data. For example, the calculated composite indicator value is 85.
[0354] 2. Store in the database: Store the obtained indicator data 85 in the metric_result table.
[0355] 3. Return the specific indicator data name and value through the interface. End: The system returns the specific indicator data name and value 85 through the interface.
[0356] Specific examples
[0357] 1. Specify the corresponding composite indicator.
[0358] 2. The system finds the following information in the composite_metric_version table:
[0359] o components: ["ta","tb","tc"]
[0360] o formula: "ta*0.5+tb*0.3+tc*0.2"
[0361] 1. The system queries the metric_result table to obtain the result data of the basic indicators ta, tb, and tc. For example, the following data is obtained:
[0362] {
[0363] "ta":100,
[0364] "tb":80,
[0365] "tc":60
[0366] }
[0367] 1. The system reads the formula ta*0.5+tb*0.3+tc*0.2.
[0368] 2. The system performs the calculation and calculates the composite metric value as 85 and stores it in the metric_result table.
[0369] 3. The system returns the indicator value 85.
[0370] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.
Claims
1. A method for dynamically configuring indicators and generating data, characterized by: The following steps are included: S1. Data Source Management Data sources include databases and external interfaces; Database management is used to manage storage connection information, and external interface management is used to manage accessible external interface information; S2. Indicator definition configuration The indicators are divided into basic indicators and composite indicators according to their value sources; basic information of basic indicators and composite indicators are configured separately; S3, indicator calculation engine The indicator calculation engine regularly discovers indicators to be updated, obtains data based on the indicator call data source, calls the script to calculate the indicator data, and saves the indicator data; The indicator calculation engine uses a polling method to obtain the indicators to be updated from the indicator library and make batch calls in the thread pool; After the indicator calculation engine calculates each indicator data and obtains the result, it changes the next update time of the indicator according to the indicator update cycle; S4, indicator data interface The indicator data interface provides indicator data to the outside world. The provided indicator data is simply aggregated according to the actual configuration; The indicator data interface needs to be configured and provide a unique call code for external calls. The configuration information includes: call code and list of aggregated indicators; According to the configuration, when the indicator data interface is called externally, it queries all indicator data in the aggregated indicator list and returns it.
2. The method for dynamically configuring indicators and generating data according to claim 1, characterized in that: The connection information in step S1 includes the connection address, user name, password, library name and other necessary parameters; The external interface information in step S1 includes the interface address, request method, request parameters, whether to paginate, pagination parameter definition, pre-script, and authentication information.
3. The method for dynamically configuring indicators and generating data according to claim 2, characterized in that: In step S1 , the database management also provides a dynamic connection and reuse mechanism for each database, that is, when a database needs to be used, if it is not connected, the connection information is obtained to connect, and if it is already connected, the connection is reused.
4. The method for dynamically configuring indicators and generating data according to claim 3, characterized in that: When defining the basic indicators in step S2, configure the indicator name, data source, single-page statistics script, data integration script, and update cycle; When defining the composite indicator, the indicator name, dependent indicator list, calculation script, and update cycle are configured.
5. The method for dynamically configuring indicators and generating data according to claim 4, characterized in that: When defining basic indicators in step S2, if the data source is a database, configure the database table, data retrieval logic and paging parameters; The statistical script obtains the data returned by the data source, passes it into the script, and executes it to obtain the final single-page statistical data; If there is paging, after all paging statistics are calculated, execute the data integration script to integrate the statistical single-page data to obtain the final data; after the indicator data is obtained, it should be saved so that it can be directly read next time.
6. The method for dynamically configuring indicators and generating data according to claim 5, characterized in that: In step S3, the basic indicators are called in different ways according to different data sources.
7. The method for dynamically configuring indicators and generating data according to claim 6, characterized in that: When the data source of the basic indicators is the database: If the basic indicator is configured with data acquisition logic, a SQL script is generated based on the data acquisition logic; if it is a SQL script, it is directly acquired; Initialize and compile the single-page statistics script for subsequent repeated calls; Get the corresponding database information and obtain the connection. If successful, assemble and execute the SQL script according to the paging situation to obtain data; The acquired data is passed into the compiled single-page statistical script to obtain the calculation results and put them into the temporary storage list; Change the parameters of the next page and assemble and execute a new SQL script to obtain data. If there is data, repeat the previous calculation action. If there is no data, jump out of the loop of the next page; The data in the temporary storage list is passed into the data integration script to obtain the calculation results, which are then stored in the indicator result table and the indicator result history table.
8. The method for dynamically configuring indicators and generating data according to claim 6, characterized in that: When the data source of the basic indicators is an external interface source: If the authentication method configured for the basic indicator is basic authentication or key authentication, it is directly used as a parameter; if the authentication method configured for the basic indicator is interface authentication, the authentication interface is called, and the returned data is used as a parameter to pass into the interface front-end script, and the paging data and the complete request object are also passed in for the script to modify; Initialize and compile the single-page statistics script for subsequent repeated calls; Use the request object modified by the script to initiate a data request and obtain the data; The acquired data is passed into the compiled single-page statistical script to obtain the calculation results and put them into the temporary storage list; Change the parameters for the next page and re-transmit them to the interface front-end script as paging data. Call the interface to obtain new data. If there is data, repeat the previous calculation action. If there is no data, jump out of the loop for the next page. The data in the temporary storage list is passed into the data integration script to obtain the calculation results, which are then stored in the indicator result table and the indicator result history table.
9. A method for dynamically configuring indicators and generating data according to claim 7 or 8, characterized in that: The composite index calculation calling method in step S3 is as follows: Obtain all basic indicators and corresponding results that the composite indicator depends on. After matching the basic indicators with the corresponding results one by one, pass them into the calculation script for calculation, obtain the calculation results, and store them in the indicator result table and the indicator result history table.
10. The method for dynamically configuring indicators and generating data according to claim 9, characterized in that: Indicator definitions can be updated and deleted; When updating the indicator definition, update the configuration information corresponding to the indicator and save the original configuration to the historical configuration table to facilitate quick query or recovery of the historical configuration; When deleting an indicator definition, only the logical deletion is performed. This logical deletion is only effective for scheduled calculations. If the query interface passes in the specified parameters, the indicator data can still be queried.
Citation Information
Patent Citations
Method and device for calculating business indexes
CN113326283A
Data index management method based on dynamic source
CN118295992A
Method and apparatus for managing data for use by data applications
US20020188584A1
Visual data analysis method and device
WO2024001493A1
