A real-time data center configuration system based on ES and its processing method
By introducing real-time data configuration module and unified data interface service module in real-time data center configuration system, using SQL and DSL syntax, JSONPath and QLEXpress technologies, the reusability and scalability of indicators displayed in the existing technology are solved. Problem, real-time data center configuration with no code adjustment and efficient development is achieved.
Patent Information
- Application Number
- CN202111269010.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-29
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-10-29
AI Technical Summary
The existing real-time data center configuration system has problems such as low reusability and scalability and poor adjustment flexibility. Developers need to repeat the steps to implement the data center display metric development for multiple services, and they cannot adjust the metric name and number without code. and computational logic.
A real-time data center configuration system based on ES was designed, including real-time data configuration module and unified data interface service module. Through the digital template configuration, simple indicator configuration, complex indicator configuration and preview submodule, the code-free adjustment indicator display is realized. Supports dynamic addition and deletion of query conditions and metrics, uses SQL and DSL syntax to generate query statements, and uses JSONPath parsing and QLEExpress expression engine to perform data processing.
It realizes the metric name, number and calculation logic of real-time data centers of each service without code, improves the reusability and scalability of indicators, reduces development costs, and improves the development efficiency and data interface coordination quality.
Smart Images

Figure CN114153866B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data center configuration systems, and in particular to an ES-based real-time data center configuration system and a processing method thereof. Background Art
[0002] With the advent of the big data era, businesses are paying more and more attention to the value of data and have higher and higher requirements for timeliness. Real-time data center dashboards have gradually become standard for every business product. Existing real-time data centers generally store real-time detailed data in the ElasticSearch cluster, add a DSL parsing engine to the ElasticSearch client layer, or parse SQL to query ElasticSearch data, perform statistics on various indicators, and then parse and convert the data results, and finally display the data on the front-end visualization interface, such as Figure 1 As shown, if business A wants to build a real-time data center, the first step is to query ES statistics for various display indicators of A; the second step is to parse the JSON string of the query results from ES to obtain simple indicators of A, such as KPI type indicators such as daily transaction volume; the third step is to customize and develop complex indicators, such as gross profit margins that require formula calculations, or complex graphic indicators such as trend charts and product lists that have complex data structures and need to be re-formatted. Finally, the simple and complex indicator data are packaged and output to the front-end data center page through the interface for display. Similarly, when business B builds its data center, it also needs to repeat the above development steps.
[0003] Therefore, the existing real-time data centers have the problems of low reusability and scalability, and poor adjustment flexibility. It is necessary to develop an ES real-time data center configuration system that can adjust the indicator name, number of indicators, indicator caliber calculation logic and other information displayed in each business real-time data center at any time without modifying the code, and can also realize indicator reuse and indicator type expansion. Summary of the invention
[0004] The purpose of the present invention is to overcome the above problems and to provide a real-time data center configuration system and a processing method for ES that can adjust the indicator name, indicator number, indicator caliber calculation logic and other information displayed in each business real-time data center at any time without modifying the code, and can also realize indicator reuse and indicator type expansion.
[0005] The object of the present invention is achieved through the following technical solutions: a real-time data center configuration system based on ES, including a real-time data configuration module and a unified data interface service module, the real-time data configuration module and the unified data interface service module are independently constructed, and the real-time data configuration module and the unified data interface service module are connected through a data line; the real-time data configuration module includes the following four submodules:
[0006] The data acquisition template configuration submodule is used to receive query conditions and replace placeholders to generate a complete query statement, and can add, edit or delete query conditions.
[0007] The simple indicator configuration submodule is used to configure the simple indicator name for the query results and parse the configured simple indicator name through the JSONPath path, and can dynamically add, edit or delete.
[0008] The complex indicator configuration submodule is used for secondary calculation script configuration of complex indicators or graphics based on simple indicators, complex indicator data calculation, and can dynamically add, edit or delete.
[0009] A preview submodule is used to preview the joint effect results after the configuration of the data acquisition template configuration submodule, the simple indicator configuration submodule, and the complex indicator configuration submodule, and present the processed results on the front-end page;
[0010] The data acquisition template configuration submodule is connected to the unified data interface service module, and the data acquisition template configuration submodule is also connected to the simple indicator configuration submodule, the complex indicator configuration submodule, and the preview submodule in sequence.
[0011] The unified data interface service module is used to connect one or more business real-time data centers with the real-time data configuration module, and the unified data interface service module can query the database configuration table for the configuration ID transmitted by the business real-time data center to generate a configuration record, can automatically generate data acquisition statements based on the configuration records, and can parse and extract the JSON strings of the data acquisition template configuration submodule, the simple indicator configuration submodule, and the complex indicator configuration submodule.
[0012] The syntax configured by the data acquisition template configuration submodule includes SQL and DSL; the data acquisition template configuration submodule supports two placeholder replacement methods: direct replacement and dynamic splicing.
[0013] The unified data interface service module is an API data service interface module or a CPI data service interface module.
[0014] The processing method of the real-time data center configuration system based on ES includes the following steps:
[0015] Step S1, one or more business data centers select one or more SQL or DSL template syntaxes for querying ES in the data retrieval template configuration submodule through the unified data interface service module, and input one or more query conditions into the data retrieval template configuration submodule.
[0016] Step S2: dynamically concatenate or replace the input query condition with the query condition configured in the data acquisition template configuration submodule to obtain a complete SQL or ES DSL query statement, and execute the query statement from the ES cluster through the obtained SQL or ES DSL query statement to obtain the query result.
[0017] Step S3, determine whether there are complex indicators or graphics in the query results. If yes, after configuring the simple indicator name for the query result obtained in step S2 through the simple indicator configuration submodule, the simple indicator configuration submodule parses the configured simple indicator name through the JSONPath path to obtain the key-value pair of the simple indicator name and its corresponding indicator value, and enters step S4; No, after configuring the simple indicator name for the query result obtained in step S2 through the simple indicator configuration submodule, the simple indicator configuration submodule parses the configured simple indicator name through the JSONPath path to obtain the key-value pair of the simple indicator name and its corresponding indicator value, and enters step S5.
[0018] Step S4: configure the complex indicator name and complex indicator name calculation expression of the complex query indicator or graph through the complex indicator configuration submodule, and inject the simple indicator key-value pair obtained in step S3 into the variable of the configured calculation expression, and obtain the complex indicator data result by executing the calculation expression.
[0019] Step S5: Present the key-value pairs of the simple indicator values obtained in step S3 and the complex indicator data results obtained in step S4 on the front-end page through the preview submodule to preview the data. After confirming that the data is correct, persist the configuration to the database, and the unified data interface service module returns the data results.
[0020] The query condition in step S1 is a key-value pair; when the query condition input in step S1 does not check "Support fuzzy query", the SQL syntax uses the in keyword and the DSL syntax uses the terms keyword; when the query condition input is checked "Support fuzzy query", the SQL syntax replaces in with like and the DSL syntax replaces terms with wildcard.
[0021] The replacement in step S2 refers to replacing the placeholders with the same name in the configuration query conditions and the data retrieval template configuration submodule. When there are multiple query conditions in step S2, the SQL syntax is spliced with and to replace the $params$ placeholder after the where condition in the data retrieval template configuration submodule, and the DSL syntax is spliced with commas to replace the $params$ placeholder in the must in the data retrieval template configuration submodule.
[0022] The dynamic splicing in step S2 includes the following two methods:
[0023] (1) The query condition name that is input and the configured query condition name are spliced as query conditions in the data acquisition template configuration submodule.
[0024] (2) If the input query condition name is different from the configured query condition name, and the configured query condition is equipped with a default value, the default value is spliced as the query condition in the data acquisition template configuration submodule.
[0025] The complex indicator name calculation expression in step S4 is the open source expression engine QLExpress, and the simple indicator key-value pairs are injected into the variables of the expression through the IExpressContext interface provided by QLExpress.
[0026] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0027] (1) The present invention uses a real-time data configuration module composed of a configuration submodule, a simple indicator configuration submodule, and a complex indicator configuration submodule to adjust the indicator name, indicator number, indicator caliber calculation logic, etc. displayed in each business real-time data center at any time. It can take effect in real time without modifying the code and going through the release and online process, and quickly respond to business needs.
[0028] (2) The present invention cooperates with the preview submodule and the unified data interface service module to achieve consistency in processing returned data and the feature that the entire set of real-time data configuration can be previewed online without persistence. After each set of business real-time data center configuration is completed, the data results can be previewed and verified in real time. After the configuration is correct, the configuration is persisted to the database. The results returned by the unified data interface service module will finally take effect, which greatly improves the quality of the interface data provided, while also reducing the cost of joint debugging of the front-end and back-end data interfaces and improving development efficiency.
[0029] (3) The present invention only needs to configure the same indicators of different business real-time data centers once, with high reusability, easy modification, and strong maintainability. When the business expands rapidly and the number of indicator types increases sharply, only new configurations need to be added, and there is no need to modify the unified data interface service, which has high scalability. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 It is a structural block diagram of the present invention.
[0031] Figure 2 It is a structural block diagram of the real-time data configuration module of the present invention. DETAILED DESCRIPTION
[0032] The present invention is further described in detail below in conjunction with examples, but the embodiments of the present invention are not limited thereto.
[0033] Example 1
[0034] like Figure 1 As shown, a real-time data center configuration system based on ES includes a real-time data configuration module and a unified data interface service module. Specifically, the real-time data configuration module and the unified data interface service module are independently constructed, which can reduce the coupling degree between the two modules. The real-time data configuration module is connected to the unified data interface service module through a data line. The real-time data configuration module is also connected to the external ES detail database. The business data center is connected to the department. In this embodiment, the unified data interface service module is simultaneously connected to the external business A data center, business B data center and business C data center. The unified data interface service module is preferably implemented by an API data service interface module or a CPI data service interface module in this embodiment. The unified data interface service module is used for the connection between one or more business real-time data centers and the real-time data configuration module, and the unified data interface service module can query the database configuration table for the configuration id passed in by the business real-time data center to generate a configuration record, can automatically generate a data acquisition statement according to the configuration record, and can extract the JSON string parsing of the data acquisition template configuration submodule, the simple indicator configuration submodule, and the complex indicator configuration submodule. When in use, different business real-time data centers pass in the configuration ID and call the unified data interface service module. The unified data interface service module finds the corresponding ES real-time data center configuration information through the real-time data configuration module based on the configuration ID. The real-time data configuration module processes the configuration information uniformly to obtain the required data and presents it on the front-end page.
[0035] like Figure 2 As shown, the real-time data configuration module includes a data acquisition template configuration submodule, a simple indicator configuration submodule, a complex indicator configuration submodule, and a preview submodule.
[0036] Specifically, the data acquisition template configuration submodule is connected to the unified data interface service module. The data acquisition template configuration submodule is also connected to the simple indicator configuration submodule, the complex indicator configuration submodule, and the preview submodule in sequence. During implementation, the preview submodule is connected to the external ES detail database and the preview submodule is connected to the external display device through the unified data interface service module.
[0037] Among them, the data retrieval template configuration submodule is used to receive query conditions and replace placeholders, generate a complete query statement, and can add, edit or delete query conditions. The syntax of the data retrieval template configuration submodule includes SQL and DSL; the data retrieval template configuration submodule supports two placeholder replacement methods: direct replacement and dynamic splicing. In specific use, when the query does not check "Support fuzzy query", the SQL syntax uses the in keyword and the DSL syntax uses the terms keyword; when the query condition input has the "Support fuzzy query" checked, the SQL syntax replaces in with like, and the DSL syntax replaces terms with wildcard. When there are multiple query conditions, the SQL syntax uses and to splice to replace the $params$ placeholder after the where condition in the data retrieval template configuration submodule, and the DSL syntax uses commas to splice to replace the $params$ placeholder in the must in the data retrieval template configuration submodule.
[0038] The simple indicator configuration submodule is used to configure the simple indicator name for the query result and parse the configured simple indicator name through the JSONPath path, and can dynamically add, edit or delete. The JSONPath path adopted by the simple indicator configuration submodule can easily obtain the content of the JSON document, thereby replacing the traditional hard-coding method to parse the ES query results.
[0039] The complex indicator configuration submodule is used for secondary calculation script configuration of complex indicators or graphics based on simple indicators, complex indicator data calculation, and can dynamically add, edit or delete. When in use, some complex indicators or graphics may need to be calculated based on ES query results or the result format conversion returns a new indicator. At this time, the complex indicator configuration submodule can configure the complex indicator name for the complex indicator, and complete the complex indicator data calculation through the calculation expression. The calculation expression in this embodiment is preferably implemented using the open source expression engine QLExpress.
[0040] The preview submodule is used to preview the joint results of the data acquisition template configuration submodule, the simple indicator configuration submodule, and the complex indicator configuration submodule after configuration, and present the processed results on the front-end page, that is, on the page of the business data center.
[0041] Example 2
[0042] The processing method of the real-time data center configuration system based on ES is characterized by comprising the following steps:
[0043] Step S1, one or more business data centers select one or more SQL or DSL template syntaxes for querying ES in the data retrieval template configuration submodule through the unified data interface service module, and input one or more query conditions into the data retrieval template configuration submodule. Specifically, the query condition is a key-value pair. When the query condition input does not check "Support fuzzy query", the SQL syntax uses the in keyword and the DSL syntax uses the terms keyword; when the query condition input has checked "Support fuzzy query", the SQL syntax replaces in with like and the DSL syntax replaces terms with wildcard.
[0044] Step S2: Dynamically concatenate or replace the input query condition with the query condition configured in the data acquisition template configuration submodule to obtain a complete SQL or ES DSL query statement, and execute the query statement from the ES cluster through the obtained SQL or ES DSL query statement to obtain the query result. Replacement means replacing the query condition with the placeholder with the same name in the data acquisition template configuration submodule. In this step, the query condition is as follows: Figure 1 As shown, when there are multiple conditions, the SQL syntax is concatenated with and to replace the $params$ placeholder after the where condition in the data retrieval template configuration submodule, and the DSL syntax is concatenated with commas to replace the $params$ placeholder in the must condition in the data retrieval template configuration submodule.
[0045] Among them, dynamic splicing includes the following two methods:
[0046] (1) The query condition name that is input and the configured query condition name are spliced as query conditions in the data acquisition template configuration submodule.
[0047] (2) If the input query condition name is different from the configured query condition name, and the configured query condition is equipped with a default value, the default value is spliced as the query condition in the data acquisition template configuration submodule. The "default value" in this step refers to a static value that does not change following the query condition, such as: 1, 2, 3······ or A, B, C, D······ and other static values.
[0048] In actual implementation, if none of the configured query condition names can match the query parameter names passed in by the unified data interface, and the configured query condition does not have a default value, the query condition will not be spliced into the data retrieval template.
[0049] Step S3, determine whether there are complex indicators or graphics in the query results. If yes, after configuring the simple indicator name for the query result obtained in step S2 through the simple indicator configuration submodule, the simple indicator configuration submodule parses the configured simple indicator name through the JSONPath path to obtain the key-value pair of the simple indicator name and its corresponding indicator value, and enters step S4; No, after configuring the simple indicator name for the query result obtained in step S2 through the simple indicator configuration submodule, the simple indicator configuration submodule parses the configured simple indicator name through the JSONPath path to obtain the key-value pair of the simple indicator name and its corresponding indicator value, and enters step S5.
[0050] Step S4, configure the complex indicator name and complex indicator name calculation expression for the complex query indicator or graph through the complex indicator configuration submodule, and inject the simple indicator key-value pair obtained in step S3 into the variable of the configured calculation expression, and execute the expression through the calculation expression to obtain the complex indicator data result. Specifically, the complex indicator name calculation expression is the open source expression engine QLExpress, and the simple indicator key-value pair is injected into the variable of the expression through the IExpressContext interface provided by QLExpress.
[0051] Step S5: Present the key-value pairs of the simple indicator values obtained in step S3 and the complex indicator data results obtained in step S4 on the front-end page through the preview sub-module, perform data preview, and persist the configuration to the database after confirming that the data is correct, and present the data results on the front-end page.
[0052] The present invention can realize the adjustment of the indicator name, indicator number, indicator caliber calculation logic, etc. displayed by each business real-time data center at any time by cooperating with the data acquisition template configuration submodule, the simple indicator configuration submodule and the complex indicator configuration submodule, and can take effect in real time without modifying the code and following the release and online process, and quickly respond to business needs. At the same time, by cooperating with the preview submodule and the unified data interface service module, the consistency of the returned data and the feature that the whole set of real-time data configuration can be previewed online without persistence can be realized. After each set of business real-time data center is configured, the verification data result can be previewed in real time, and the configuration will be persisted to the database after it is correct. The unified data interface service module will return the result and it will finally take effect, which greatly improves the quality of the interface supply data, and also reduces the cost of joint debugging of the front-end and back-end data interfaces, and improves the development efficiency.
[0053] As described above, the present invention can be well implemented.
Claims
1. A real-time data center configuration system based on ES, It is characterized in that It includes a real-time data configuration module and a unified data interface service module, which are independently constructed and connected via a data line; the real-time data configuration module includes the following four submodules: The data acquisition template configuration submodule is used to receive query conditions and replace placeholders, generate a complete query statement, and can add, edit or delete query conditions; The simple indicator configuration submodule is used to configure the simple indicator name for the query result and parse the configured simple indicator name through the JSONPath path, and can dynamically add, edit or delete; Complex indicator configuration submodule, used for secondary calculation script configuration of complex indicators or graphics based on simple indicators, complex indicator data calculation, and dynamic addition, editing or deletion; A preview submodule is used to preview the joint effect results after the configuration of the data acquisition template configuration submodule, the simple indicator configuration submodule, and the complex indicator configuration submodule, and present the processed results on the front-end page; The data acquisition template configuration submodule is connected to the unified data interface service module, and the data acquisition template configuration submodule is also connected to the simple indicator configuration submodule, the complex indicator configuration submodule, and the preview submodule in sequence; The unified data interface service module is used to connect one or more business real-time data centers with the real-time data configuration module, and the unified data interface service module can query the database configuration table to generate configuration records for the configuration ID passed in by the business real-time data center, can automatically generate data retrieval statements based on the configuration records, and can parse and extract the JSON strings of the data retrieval template configuration submodule, the simple indicator configuration submodule, and the complex indicator configuration submodule.
2. According to the ES-based real-time data center configuration system of claim 1, It is characterized in that The syntax configured in the data acquisition template configuration submodule includes SQL and DSL; the data acquisition template configuration submodule supports two placeholder replacement methods: direct replacement and dynamic splicing. The direct replacement refers to replacing the placeholder with the same name as the configuration query condition in the data acquisition template configuration submodule; when there are multiple query conditions, the SQL syntax is spliced with and to replace the placeholder after the where condition in the data acquisition template configuration submodule. Placeholder, DSL syntax is concatenated with commas to replace the must in the data retrieval template configuration submodule Placeholders; The dynamic splicing includes the following two methods: (1) if the input query condition name is the same as the configured query condition name, it is spliced into the data acquisition template configuration submodule as the query condition; (2) if the input query condition name is different from the configured query condition name and the configured query condition is equipped with a default value, the default value is spliced into the data acquisition template configuration submodule as the query condition.
3. According to the ES-based real-time data center configuration system of claim 2, It is characterized in that The unified data interface service module is an API data service interface module or a CPI data service interface module.
4. A processing method for a real-time data center configuration system based on ES according to any one of claims 1 to 3, It is characterized in that The following steps are involved: Step S1, one or more business data centers select one or more SQL or DSL template syntaxes for querying ES in the data acquisition template configuration submodule through the unified data interface service module, and input one or more query conditions into the data acquisition template configuration submodule; Step S2: dynamically concatenate or directly replace the input query condition with the query condition configured in the data acquisition template configuration submodule to obtain a complete SQL or ES DSL query statement, and execute the query statement from the ES cluster through the obtained SQL or ES DSL query statement to obtain the query result; Step S3, determine whether there are complex indicators or graphics in the query result. If yes, after configuring the simple indicator name for the query result obtained in step S2 through the simple indicator configuration submodule, the simple indicator configuration submodule parses the configured simple indicator name through the JSONPath path, obtains the key-value pair of the simple indicator name and its corresponding indicator value, and enters step S4; No, after configuring the simple indicator name for the query result obtained in step S2 through the simple indicator configuration submodule, the simple indicator configuration submodule parses the configured simple indicator name through the JSONPath path, obtains the key-value pair of the simple indicator name and its corresponding indicator value, and enters step S5; Step S4, configure the complex indicator name and complex indicator name calculation expression for the complex query indicator or graph through the complex indicator configuration submodule, and inject the simple indicator key-value pair obtained in step S3 into the variable of the configured calculation expression, and obtain the complex indicator data result by executing the calculation expression; Step S5: Present the key-value pairs of the simple indicator values obtained in step S3 and the complex indicator data results obtained in step S4 on the front-end page through the preview submodule to preview the data. After confirming that the data is correct, persist the configuration to the database, and the unified data interface service module returns the data results.
5. The processing method of the ES-based real-time data center configuration system according to claim 4, It is characterized in that The query condition in step S1 is a key-value pair; when the query condition input in step S1 does not check "Support fuzzy query", the SQL syntax uses the in keyword and the DSL syntax uses the terms keyword; when the query condition input has checked "Support fuzzy query", the SQL syntax replaces in with like and the DSL syntax replaces terms with wildcard.
6. The processing method of the ES-based real-time data center configuration system according to claim 4, It is characterized in that The complex indicator name calculation expression in step S4 is the open source expression engine QLExpress, and the simple indicator key-value pairs are injected into the variables of the expression through the IExpressContext interface provided by QLExpress.
